Peer Review under the Spotlight
ISiS Commentary, 20 May 2008
http://www.i-sis.org.uk:80/peerReviewUnderTheSpotlight.php

What matters most is the lack of public scrutiny rather than the lack of peer review in times of corporate corruption of science Prof. Peter Saunders

You've probably come across the expression "peer-reviewed" a lot recently, especially in discussions on GM food, mobile phones, or organic farming. It's almost always used as part of a sentence that begins "There is no peer-reviewed evidence for ”¦" or "There is nothing about this in any peer-reviewed journal ”¦"

What you're meant to understand by that is: "there is no credible evidence for whatever it is, and you can safely ignore anything you've heard about it." When the question is about safety, as it often is, it means the regulatory authorities are not going to look into it.

A lot of people take peer review very seriously, or at least they say they do. When Sir David King, then the Chief Scientific Adviser to the British government, put forward a code of ethics for scientists, one of his chief examples of unethical behaviour, right up there with plagiarism, was "disseminating work before it has been peer reviewed". You may remember Arpad Pusztai who spoke for 150 seconds in a television programme on unpublished results indicating that genetically modified (GM) potatoes were harmful to rats, because he saw it his duty to warn the public. He was subjected to fierce attacks from the scientific establishment (led by the Royal Society) that continue to the present day.

The scientific establishment's double standard

The scientific establishment may claim to oppose disseminating results that have not been peer-reviewed, but there is a blatant double standard being applied, and all too often. Just recently, a UK government research funding agency, the Economic and Social Research Council (ESRC) put out a press release that was not only highly misleading about farmers being upbeat about GM crops ("UK Farmers Upbeat about GM Crops" Debunked and Marketing Masquerading as Scientific Survey , SiS 38); but was also based on research that had not been peer-reviewed, according to the ESRC's own web site.

Another recent example came from the top mainstream journal Nature Biotechnology. In an editorial, it criticised the Italian National Research Institution for Food and Nutrition (INRAN) for not publishing some results that were allegedly favourable to GM crops.

The director of the project in question, Giovanni Monasatra, wrote to the journal to put the story straight, and his letter was published along with a response from the editor, Andrew Marshall. In his letter, Monastra dealt with the points raised in the editorial and expressed his surprise that Marshall, far from criticising Salute, Agricolura, Ricerca (SAGRI) for organising a press conference to publicise data that were, according to Marshall, "too preliminary for peer-reviewed publication," instead complained that the Italian media did not give it even more coverage than they did.

Marshall's response is that the data had [his italics] to be press released by SAGRI because they were of interest to the public and political debate. Yet, in 1999, Marshall had written in Nature Biotechnology that Arpad Pusztai's work should be submitted for peer review before it could be considered, even when safety was at stake. The difference is that he was then writing about results that were against the interests of the biotech industry.

It is not at all unusual for scientific bodies and lobby groups to issue press releases and hold press conferences on non-reviewed material. The people who set themselves up as the guardians of sound science either say nothing or even join in, except when it is a matter of things the corporations don’t want the public to hear. In that case, they suddenly rediscover their strong objection to the practice.

Peer review is a useful part of the scientific process, but it is not as effective, as important, or as universal as some would have us believe, and it needs to be put in perspective.

What is peer review?
One of the distinguishing features of science is that when you discover something, you don’t expect other people just to take your word for it. You’re expected to describe exactly what you have done and why this justifies what you claim, and the usual way of doing this is to publish a paper in a scientific journal.

When you submit your paper to a journal, the editor will generally send it to be reviewed by experts in the field. The referees, usually two or three, are supposed to read the manuscript carefully and assure themselves as best they can that the work was done using appropriate techniques, that it takes into account and properly acknowledges earlier relevant work, and that the conclusions are properly derived from the data, or, in the case of a theoretical paper, that the arguments are sound. They advise on whether the work is interesting enough and contains enough that is new to be worth publishing, and, even if it is, whether they consider it is suitable for the particular journal. They may also suggest ways in which the paper could be improved.

This peer review system is important in science. It prevents many very poor papers from being published and it improves many others. Above all, it helps maintain a consensus of what is expected in a scientific paper; what you find in a scientific journal is very different not only from the popular press but even from most papers in the humanities or social sciences.

But peer review is very limited in what it can do. Referees, who are not paid, vary considerably in the time and effort they devote to the task. They are all too likely to nod a paper through if it looks plausible and comes from a lab or a group that they know and trust, or to reject one because they disagree with it or don’t understand it and haven’t the time or inclination to go through it carefully. They may reject a paper as “not interesting” when what they mean is that it’s not the sort of thing they and their friends are interested in.

Referees do not go into the laboratory to watch the experiments being carried out. They do not have access to the authors’ notes and raw data. They make their decisions on the basis of nothing more than what the reader will see if the paper is accepted. Even the most conscientious are simply not in a position to guarantee that the results are correct or even that the work was done properly.

Peer review could certainly be improved, and there are a number of ways in which this might be done. For example, research has confirmed the suspicion of many scientists that there is often bias, whether conscious or not, and it has been suggested that referees should not be told the authors’ names or institutions, or even their gender. But while the system could be improved, it is hard to see how it could do much more, even if we really want it to. The real test of a paper comes after it is published and is open for comment by the whole of the scientific community, and that can be a far more stringent test. Indeed, many poor or outright fraudulent works have been exposed after they were in print. One recent example is a paper published in the British Food Journal and given an Award for Excellence, which provoked 40 scientists and two MPs to sign an open letter demanding its retraction (Wormy Corn Paper Must be Retracted, SiS 37). We shouldn’t expect peer review to do more than it actually can, and by the same token we shouldn’t claim to the public that it does.

Not all science is peer reviewed
If all real science were peer reviewed before it was made public or used in decision making, then you might want to say that peer review, rightly or wrongly, defines real science. In fact, there are a number of ways in which science often gets into circulation without peer review. For example, the work can be published in the proceedings of scientific meetings or, especially in rapidly moving subjects like theoretical physics, circulated as a preprint or posted on a web server.

On the whole, exceptions like those don’t matter much. Most of the work will eventually be refereed or else just forgotten. In any case, it is out in the open for anyone to see and criticise.

The absence of peer-review matters in regulation but not as much as the absence of public scrutiny
There is, however, an area in which the absence of peer review matters a great deal, and that is in regulation. Many products, including pharmaceuticals and GM foods, have to be licensed. The manufacturers are required to carry out trials, safety tests and risk assessments and submit the results to the regulators.

Much of this work is never peer reviewed. What is more, much of it is never published, and worse, concealed from the public and often from the regulators as well under claims of “commercial confidentiality”, so that other scientists are never able to comment on it.

Companies use commercial confidentiality in much the same way that the UK government uses the Official Secrets Act, less as a means of keeping sensitive information from a possible competitor than to ensure that nothing embarrassing reaches the public. And as with the Official Secrets Act, the claim that something must be kept confidential on commercial grounds is seldom challenged. Just to quote one example, even after the TGN1412 trial went so disastrously wrong (see Post Mortem on the TGN1412 Disaster, SiS 30), and it was abundantly clear that the drug would never be developed further, the Medicines and Healthcare Products Regulatory Agency (MHRA) still refused to release some details of the test protocol on the grounds of commercial confidentiality. It is hard to imagine how a competitor of either Te Genero, the company that developed the drug, or Parexel, the company that ran the trials, could have gained any unfair advantage from the information, though it might have been of use to anyone trying to improve the safety of trials, and more importantly, to provide effective remedy for the victims.

The absence of peer review is nowhere near as important as the absence of public scrutiny. To make claims on the basis of research that you will not reveal to the scientific community is to go against one of the basic principles of science: that we provide evidence for the claims we make. It is all the more serious when these claims can affect health and the environment. Such data should never be held secret on grounds of commercial confidentiality.

Conclusion
Peer review is a useful part of science but it is not and cannot be the dividing line between good science and bad, between what can be relied upon and what must be dismissed out of hand. In particular, when we are told that there is no peer-reviewed evidence, that does not mean that there is no evidence, nor does it mean there is no credible evidence. The scientific establishment has been deliberately applying a double standard to exclude evidence unfavourable to industry. What is worse, our regulators have accepted all kinds of evidence in approving new products and processes not just without peer review, but without the possibility of scrutiny by the public or even by the regulators themselves.

A fully referenced version of this article is posted on ISIS members' website.

An electronic version of this report, or any other ISIS report, with full references, can be sent to you via e-mail for a donation of GBP3.50. Please e-mail the title of the report to: This email address is being protected from spambots. You need JavaScript enabled to view it.