Recent news reports have observed that a lot of published scientific papers are worthless and not true, with initial claims turning out to be false. This is particularly worrisome where newly invented medicines are concerned. The problem also affects published results in fields as diverse as psychology and global warming. We ask “what’s wrong?” The problem seems to be that the system known as “peer-review” is not doing its job properly.
The process of peer review is a cornerstone of scientific integrity, the guarantee of quality in a scientific research paper. Catch phrases like “holy grail” come to mind. When a researcher accomplishes something, the concluding step is to publish the results in the scientific literature. Enroute to publication, the paper must undergo peer review.
The author(s) of the scientific paper decides to send it to a particular journal for consideration, and then the peer-review process begins. The editor sends the paper to other scientists who are the peers (equally-capable scientists) of the author, and active in the same research area. Upon reading the paper, each such reader sends back to the editor a review with comments and suggestions for improvement of the paper. The reviewers remain anonymous relative to the author, to avoid softening of criticism that might be due.
Based on that input, the editor either accepts the paper for publication, sends it back to the author for revision, or rejects it outright.
Most of us professional scientists have experienced all three outcomes at various times. By far the most common is to be asked to clarify some technical points, or to pay attention to some overlooked paper in the literature that has bearing on the topic. After revising the text to conform to the criticism of the reviewers, the paper customarily goes forward for publication.
Because of that process, another scientist reading the journal has confidence that the papers published in it are reliable, and subsequently uses that information to influence the direction of his or her own research. The peer review process is the best way known to scientists to advance their professional disciplines. Without the checks and balances of such a process, the scientific literature would deteriorate into just another news outlet, a cluster of press-releases and advertisements.
Most academics are evaluated on the basis of their published papers, and often a major criterion of being promoted is a count of the number of peer-reviewed papers a professor has. That translates into both prestige and salary, and hence there is built-in pressure upon a junior faculty member to generate many papers. The slogan “publish or perish” is well known in academia. This practice holds across all academic disciplines, including history, psychology, social sciences, literature — not just sciences like chemistry, biology or physics.
Some journals are more prestigious than others, and there is strong competition to have a paper published in a more prestigious rather than in a lesser journal. For example, the hardest journal to get a physics paper accepted into is Physical Review Letters, and a professor who gets even one paper published there during his/her whole career is very proud of it. Biochemistry, medicine, geology, history, and so on each have a hierarchy of journals. The New England Journal of Medicine is America’s most prestigious medical journal.
The public is oblivious to all this, and only hears about peer review as the certification of validity among scientists. The perception is ironclad that a community of scientists is self-disciplining, and truth will surely emerge from the process over time. Nearly all government agencies share this perception, and courts of law routinely defer to their judgment on scientific questions, never examining the underlying research.
For the peer-review system to work effectively, everybody in the loop has to be scrupulously honest. The authors must have reached their conclusions honestly, based on valid data; and be striving to present new information accurately to readers. The editor must select unbiased reviewers to conduct the peer-review. The reviewers must be objective, guarding against letting their own biases creep into their evaluations. When all this happens, a paper eventually gets published that adds an incremental advance to the field.
However, human beings are imperfect, and hence the peer-review process doesn’t always work ideally. The very essential elements of integrity and trust can easily be lost at several stages. Authors are sometimes blind to their own mistakes; and peer-reviewers can be variously misinformed, hostile, or motivated by hidden agendas. Editors might have their own narrow views about what they want in their journal; they can select certain reviewers to obtain negative or positive evaluations. When things go wrong, bad information gets unduly promoted. The history of peer review is not unblemished.
