At the end of June the lobby group Sense About Science finally issued the report of its Working Party on peer review. The report and an acompanying press release can be found here: http://www.senseaboutscience.org.uk/peerreview/index.htm
An article about the report (see below) from the Education Guardian states, "The public and the media could expose wild and bogus scientific claims if they asked tougher questions about the status of research, a working party of leading scientists said today."
Sense About Science is a controversial organisation and its Working Party on peer review has attracted criticism since its inception. At an early stage the Wellcome Trust, well known for its generous support of work on the public understanding of science, set out in a letter to Sense About Science why, after careful consideration, it was declining to be part of the Working Party which drew up this report or to provide any funding for its work. Among the reasons the Wellcome Trust gave were: "The proposed make-up of the Working Party is extremely narrow" and "runs the risk of being seen as a closed and defensive strategy", and the letter talks of the project being based on "many assumptions" and very little "direct evidence".
The "extremely narrow" group which made up the Working Party included 3 members of the ideologically extreme LM network (Fiona Fox, Tony Gilland and Sense About Science's Director, Tracey Brown, who wrote up the Working party's report) as well as the former Vice President of the Royal Society, Brian Heap, and the Society's former Biological Secretary, Peter Lachmann. Liaising with the group was the Royal Society's media and PR man, Bob Ward.
Early press comment suggested that this report and a parallel report expected from the Royal Society itself would be aimed at issues like GM and the work of researchers like Dr Arpard Pusztai. The report does not disappoint in this respect, containing multiple general references to GM along with MMR, mobile phone radiation and other "scares", as examples of concerns not based on rigorous peer reviewed research. Interestingly, however, in relation to GM not a single specific example of this problem appears to be given.
The report does contain a specific, though indirect, reference to Stanley Ewen and Arpad Pusztai's research on feeding GM potatoes to rats. As the form of this reference reveals a lot about the character and agenda of those pushing forward this Sense About Science project, it is worth quoting the point in the report that contains the reference in full:
"3.6 Some science commentators have tended to present complaints about peer review without first explaining why it is used. In 1999, Richard Horton, editor of The Lancet, described peer review as "usually ignorant" and "frequently wrong"57 after the journal was criticised for publishing research on the effects of feeding GM potatoes to rats, which had been turned down by another leading journal. He did not, though, explain why The Lancet would continue to base its publication decisions on reviews that are "usually ignorant". Horton also argued that BSE had made the public deeply sceptical about science and that The Lancet was encouraging a more open debate by publishing the GM paper. Being vague as to whether decisions about scientific publication should be based on peer review or on editorial ideas about the needs of public discussion is unlikely to diminish public scepticism, or to promote well-informed debate, and indeed it does not appear to have done either."
[The reference [57] is to: Horton (2000), Genetically modified food: consternation, confusion and crack-up. The Medical Journal of Australia, 172: 148-149. http://www.mja.com.au/public/issues/172_04_210200/horton/horton.html ]
The report gives the impression that because Ewen and Pusztai's research had, as it claims, already been rejected by "another leading journal", Horton was forced to defend his own publication of the research by attacking the value of peer review and suggesting publication was in the interest of public debate, ie the issue of scientific merit had been set aside.
The clear inference of this is that the Lancet's publication of Ewen and Pusztai's research wasn't based on normal peer review criteria. This, however, could not be further from the truth:
1. Far from being rejected by "another leading journal", the Pusztai research was only ever submitted to The Lancet.
2. The editor of The Lancet not only subjected the research to peer review, he subjected it to an especially stringent version, sending it to double the normal number of referees.
3. In the article by Horton to which the Working Party's report refers, Horton explicitly states that, "five out of six of The Lancet's reviewers judged that Ewen and Pusztai's work should be published".
4. Although Horton has spoken of the importance of the Pusztai paper's publication in relation to open public debate, he has also made it clear that, "Stanley Ewen and Arpad Pusztai's research letter was published on grounds of scientific merit, as well as public interest".
We asked Arpad Pusztai for his comments on the Working Party's claims. Pusztai comments, "They keep up their noble tradition and prove that even in a short para they can be just as untruthful as in their much longer reviews and reports about me. As usual, you were right and I can confirm that we always intended our paper for the Lancet and have NEVER submitted it to another major or minor journal."
Interestingly, Horton in the article referred to by the Working Party, far from defending his decision to publish the Pusztai research against the refusal to publish of "another leading journal", in fat contrasts the support for publication of a clear majority of The Lancet's reviewers with the results of a partial "peer review" of Pusztai's then unpublished research which was undertaken by a group organised by the Royal Society.
The Royal Society's review was based not on a properly prepared paper, like that which Pusztai and his collaborator Ewen submitted for review to The Lancet, but rather on a far-from-complete internal report intended for use by Pusztai's research team at the Rowett Institute. Elsewhere, Horton has described the Royal Society review as "a gesture of breathtaking impertinence to the Rowett Institute scientists who should be judged only on the full and final publication of their work."
The Royal Society has not, however, been in the business of waiting for publication before asserting its views. In fact, there are clear indications that its intention has always been the exact opposite.
In February 1999, for instance, nineteen Fellows of the Royal Society condemned Pusztai, in all but name, in a letter published in the national press. Among the signatories was Peter Lachmann who sat on the Sense About Science Working Party which drew up this report.
In May 1999 the Royal Society published the partial "peer review" of Pusztai's unpublished research. After Richard Horton publicly criticised the Society's behaviour, Lachmann responded with a letter to The Lancet, attacking both The Lancet and the British Medical Association for "aligning" themselves "with the tabloid press in opposition to the Royal Society and Nuffield Council on BioEthics".
The Royal Society's partial review was organised by members of a working group appointed by the Society in coordination with the Society's officers who included Brian Heap. Heap was also part of the Nuffield Council on BioEthics and he chaired the Sense About Science Working Party which produced this report.
Prior to The Lancet's publication of the Ewen and Pusztai paper, Richard Horton received a phone call from Peter Lachmann. According to Horton, Lachmann called him "immoral" for publishing something he knew to be "untrue". Towards the end of the conversation Horton says Lachmann also told him that if he published Pusztai's paper, this would "have implications for his personal position" as editor. The Guardian broke the news of Horton being threatened in November 1999 in a front-page story. It quoted Horton saying that the Royal Society had acted like a Star Chamber over the Pusztai affair. "The Royal Society has absolutely no remit to conduct that sort of inquiry". Lachmann denied threatening Horton although he admitted making the phone call in order to discuss the pending publication.
According to the article below, the Sense About Science Working Party, containing Peter Lachmann, are saying scientific claims "should be challenged" to see if they have been properly peer reviewed. But its clear that there are members of this Working Party who are interested in challenging what they regard as "problematic" scientific research regardless of its standing and are quite happy to make questionable claims as a means of doing so. Indeed this report exemplifies this by itself containing bogus claims.
A GM WATCH profile of the Royal Society is available here:
http://www.gmwatch.org/profile1.asp?PrId=113
A GM WATCH profile of Sense About Science is available here:
http://www.gmwatch.org/profile1.asp?PrId=151&page=S
A GM WATCH profile of Sense About Science Working Party member Peter Lachmann is available here:
http://www.gmwatch.org/profile1.asp?PrId=74
A GM WATCH profile of Working Party chair Brian Heap is available here:
http://www.gmwatch.org/profile1.asp?PrId=55&page=H
A GM WATCH profile of Working Party member Fiona Fox is available here:
http://www.gmwatch.org/profile1.asp?PrId=45
A GM WATCH profile of Working Party member and Director of Sense About Science, Tracey Brown is available here:
http://www.gmwatch.org/profile1.asp?PrId=143
A GM WATCH profile of Working Party member Tony Gilland is available here:
http://www.gmwatch.org/profile1.asp?PrId=61&page=G
A GM WATCH profile of the LM network is available here:
http://www.gmwatch.org/profile1.asp?PrId=78
------
Scientific claims 'should be challenged'
Donald MacLeod
Education Guardian, June 24, 2004
http://education.guardian.co.uk/higher/research/story/0,9865,1246476,00.html
The public and the media could expose wild and bogus scientific claims if they asked tougher questions about the status of research, a working party of leading scientists said today.
The group, including Professor Colin Blakemore, head of the Medical Research Council, wants to educate the public about peer review - the process by which scientific findings are assessed by experts in the same field before being published.
The scientists believe that if reporters asked if claims about, for example, genetically modified crops or the MMR vaccine had been peer reviewed before being published or broadcast there would be fewer scare stories and the public would be better able to weigh up conflicting theories. Flawed and implausible claims would be shown up because they had failed the test of scientific opinion, they believe.
Peer review is second nature to scientists but, they admit, almost completely unknown to the general public. The working party, convened by the Sense About Science trust, today urged scientists to explain the process better whenever they have a chance.
Every year, an estimated 1 million papers about the findings of scientific research are published after being assessed for competence, significance and originality by scientists publishing work in the same field. Many weak or flawed papers are rejected. Others are criticised, improved and published.
"This process of 'peer review' is a much more reliable guide to whether findings are plausible than who conducted the work or how it was funded. But scientists have rarely explained that peer review is used to decide which research is worthy of publication - and very few people outside the science world know to ask whether research has been peer reviewed," said the working party.
Tracey Brown, director of Sense About Science and a member of the working party, said: "The paper is proposing a simple cultural shift towards wider knowledge of the peer-review process, so that all people with an interest in scientific issues start asking tough questions about the information that is put before them, questions like: Have these research claims been peer reviewed? Has the study been published in a recognised scientific journal? And, how many other research papers have reached the same conclusions?"
She added: "It is well-recognised that it's easy to generate publicity, concern or political attention on the back of unsubstantiated research claims. When the working party reviewed the contentious claims of recent years, relating to the MMR vaccine, mobile phones, GM crops and transgenic animals among others, it was surprised that so little had been said about how scientific peers had assessed the claims and, in some cases, why the research had not been shown to those peers before the results were made public. That information is crucial for non-experts weighing up the claims and people should know to ask for it.
"As most scientists will tell you, peer review is by no means the last word on a piece of research, but a culture of explaining and asking about peer review all along the line - from radio phone-ins to ministerial briefings - will put a lot more pressure on people bringing research claims to the public to explain exactly what the status of the work is."
The paper, Peer review and the acceptance of new scientific ideas, which includes a guide to the peer review process, and discusses issues including anonymity, editorial bias, maverick science, commercial research and conflicts of interest, can be downloaded from www.senseaboutscience.org