UN’s Climate Bible Gets 21 ‘F’s on Report Card

• all 18,531 references cited in the 2007 IPCC report were examined
• 5,587 are not peer-reviewed
• IPCC chairman’s claim that the report relies solely on peer-reviewed sources is not supported
• each chapter was audited three times; the result most favorable to the IPCC was used
• 21 out of 44 chapters contain so few peer-reviewed references, they get an F
• 43 citizen auditors in 12 countries participated in this project
• full report card here
• detailed results here

Citizen Audit Main Findings
released April 14, 2010

BACKGROUND AND INTRO
United Nations countries belong to an organization called the Intergovernmental Panel on Climate Change (IPCC), which publishes a report every six years. Often referred to as the “climate bible” the latest one was released in 2007 and is relied on by governments around the world. Billions of dollars are spent on national and international policies based on its findings. Judges consult it when trying cases. Scholars and journalists cite it thousands of times a year.

The IPCC report contains 44 chapters and is nearly 3,000 pages long. Written by people organized into three teams – Working Group 1, 2 and 3 – it consists of three smaller reports bundled into one.

PEER-REVIEWED LITERATURE CLAIM
The chairman of the IPCC has declared repeatedly that the report is based solely on peer-reviewed literature. (This means research papers that have been submitted to an academic journal, scrutinized by anonymous referees, and frequently altered in order to qualify for publication. Although the peer-review process does not guarantee accuracy, the fact that research findings have undergone this process promotes a feeling of confidence.)

This Citizen Audit focused its attention on the peer-reviewed literature claim. A team of 43 volunteers from 12 countries examined the list of references at the end of each chapter. We sorted these references into two groups – articles published in peer-reviewed academic journals and other references. (Non-peer-reviewed material is often called “grey literature”.) Then we calculated the percentage of references that do, indeed, appear to be peer-reviewed.

In elementary schools in the United States, students are assigned grades ranging from an A to an F, based on the mark they’ve achieved out of 100 (see Wikipedia’s table here). Most parents would be alarmed if their child brought home a report card similar to the one received by the IPCC.

21 out of 44 chapters contain so few peer-reviewed references that the IPCC received an F. The IPCC relied on peer-reviewed literature less than 60 percent of the time in these chapters.

5,587 references in the IPCC report were not peer-reviewed. Among these documents are press releases, newspaper and magazine articles, discussion papers, MA and PhD theses,

working papers, and advocacy literature published by environmental groups.

Citizen Audit graphic                                                   IPCC graphic

WHY THESE FINDINGS MATTER
Governments around the world need to base their policies on impeccable research – not a report that relies on 5,587 instances of grey literature to make its case. If individuals with an agenda had wished to manipulate this report, they were afforded thousands of opportunities to do so.

Rajendra Pachauri, the IPCC chairman, has repeatedly trumpeted his organization’s full reliance on peer-reviewed literature. But the report falls well short of that standard. If the IPCC has misled the public about a matter this straightforward surely its ability to accurately describe more complex topics is called into question.

OUR METHODOLOGY
Most chapters in the IPCC report contain hundreds of references. Not every reference is adequately documented, and classifying some as peer-reviewed or gray literature involves a measure of discretion. Our How-To Guide advised auditors to: “Give the IPCC the benefit of the doubt.”

Each chapter was examined by three citizen auditors working independently of each other (often on different continents). On those occasions in which the auditors arrived at slightly different results, the span between the highest and lowest result does not exceed five percent. We used the result most favorable to the IPCC to calculate our findings. [more about our quality assurance mechanisms]

WHO WE ARE
This project was administered by Canadian blogger Donna Laframboise, author of the upcoming book Decoding the Climate Bible. On March 8, she invited citizen volunteers to help audit the 2007 IPCC report. More than 40 people participated in this crowd-sourcing project, collectively donating hundreds of hours of their personal time. The last audit was completed on April 7.

These citizen auditors live in different parts of North America but also in Australia, Costa Rica, Cyprus, Finland, Germany, New Zealand, South Africa, Sweden, The Netherlands, and the UK. Among them are a medical doctor, a lawyer, and people with degrees in engineering, chemistry, geophysics, mathematical physics, and plasma physics. Others have commerce, economics, and arts degrees. Some are retired. Nine are women.

Source