NOTE: We recently published comments by a former government research analyst and statistics expert about the attacks on Prof Seralini's paper, which showed increased tumours, premature deaths and liver and kidney pathology in rats fed Monsanto's GM maize NK603 and also tiny amounts of Roundup.
http://www.gmwatch.org/index.php?option=com_content&view=article&id=14249
Many of the criticisms, which were gathered and disseminated by the Big Pharma- and Big GM-industry-funded Science Media Centre, have focused on the statistical analysis and interpretation of the findings.
The Science Media Centre has published an analysis of Seralini's paper (item 1 below), produced by its own team of stats experts:
http://www.sciencemediacentre.co.nz/2012/09/20/study-on-cancer-and-gm-maize-experts-respond/
Their list of experts and their day jobs reads like a Who's Who of Big Pharma/Big Biotech (the pharmaceutical industry has significant crossover with the biotech industry):
http://www.sciencemediacentre.org/pages/press_releases/before_the_headlines.htm
We sent the SMC team's comments to the former government research analyst. The analyst came back with responses.
We've given the SMC comments as a whole in item 1 below. Then in item 2 below we've broken up the SMC's comments into parts, each preceded with "SMC", and followed them with the analyst's comments, again clearly marked.
–-
–-
1.Before the Headlines Commentary
Science Media Centre, September 19 2012
http://www.smc.org.au/2012/09/rapid-reaction-long-term-toxicity-of-gm-maize-food-and-chemical-toxicology-experts-respond/
Below is an analysis by UK statisticians assessing the validity of the findings in the paper and comments from UK experts from our colleagues at the UK Science Media Centre.
Title, Date of Publication & Journal
'Long term toxicity of a Roundup herbicide and a Roundup-tolerant genetically modified maize', Food and Chemical Toxicology 19 September 2012
Claim supported by evidence?
The paper does not prove the claim that rats fed on a diet containing NK603 Roundup tolerant GM maize died earlier than rats fed on a standard diet.
Summary
Looking at the graphs of mortality for females (Fig 1), death might appear to be earlier with a GM diet than a standard diet; however this has not been proven statistically. Mortality is broadly similar for males with a GM diet and a standard diet. Similar comments apply to pathological findings.
It is evident that some treated groups have lower death rates / tumour rates than the comparable controls. This is not reported in the abstract.
There is no consistent dose-trend if there were an effect, we would expect to see increases (e.g. in deaths) from 0 to 11 to 22 to 33. In contrast – in males, 33 (and C) have the lowest numbers of deaths.
Strengths/Limitations
"In females, all treated groups died 2-3 times more than controls, and more rapidly. This difference was visible in 3 male groups fed GMOs." – this statement has not been subjected to standard methods of statistical analysis for survival time. The phrase "died 2-3 times more than controls" is based on exceptionally small numbers.
The authors suggest a threshold – this rarely occurs in practice. We would expect greatest mortality/ toxicity at the highest dose in a well-designed study. With small numbers as in this study we would expect to see a general trend of mortality increasing with dose.
It seems likely that the numbers in each group are too small for standard methods of statistical analysis to find significant effects on mortality or pathological findings.
There are virtually no p-values presented. The group sizes are small. This should be interpreted with extreme caution.
There are many treated groups, and a number of parameters. There is obvious potential for selected reporting, selection of methods etc. In such small groups, with so many parameters this is a big issue. This issue is amplified in the abstract and further in the press release. Strong statements are issued without sufficient backing / explanation.
Deaths are compared to the control mean (for males and females). Due to the distribution of deaths (most deaths occur in old age), this is almost bound to exclude the large majority of deaths in the control groups. The 2 or 3 deaths in the control group is determined by their methods, but is inappropriately presented as a true observation.
–-
–-
2. Research analyst's comments on Science Media Centre's Commentary
Published by GMWatch, 3 Oct 2012
SMC: The paper does not prove the claim that rats fed on a diet containing NK603 Roundup tolerant GM maize died earlier than rats fed on a standard diet.
Analyst's comment: Where did anyone claim proof? The point was the two-year duration had results suggestive of effects greater than in the controls. Statistical tests don't "prove" anything (that's pharma industry talk). They test the hypothesis and either support it or reject it using some significance level of p.
SMC: Looking at the graphs of mortality for females (Fig 1), death might appear to be earlier with a GM diet than a standard diet; however this has not been proven statistically.
Analyst's comment: In fact, the graphs do show this – in numbers, euthanasia need (tumour progression), and timing. If these critics are statisticians why didn't they suggest a method and do some examples to show us if their criticism has any legs. I agree the data and hypothesis have not been tested (again proof is the wrong word). I also note that they used the word "appear" in this comment on females, but the word "evident" for the comment below on males.
SMC: Mortality is broadly similar for males with a GM diet and a standard diet. Similar comments apply to pathological findings.
Analyst's comment: The mortality difference is not as large as for females, but there are still differences that I would not dismiss as "broadly similar". What does that mean? They should be tested.
They also state that the pathology (Table 2, not mentioned by them) was similar, but this is clearly not the case for females. The pathology for females is clearly different. But they don't mention this, and their statement could be seen as applying generally to both sexes.
As statistician critics they had the chance to show examples of support for their comments. I think for the females, when 10 out of 10 treatments show pathology, compared to 5 controls, this is significant using the Fischer exact test. Everything should be tested, not just eyeballed for or against.
SMC: It is evident that some treated groups have lower death rates / tumour rates than the comparable controls. This is not reported in the abstract.
Analyst's comment: This could be written more exactly, which makes it misleading, as is using the words "evident that some". In females, no treatment group was lower for mortality or tumours than controls. Treatments showed more progression and euthanasia than controls for mortality, and more spontaneous mortality. Two female treatments were the the same as controls for all tumours, but all treatments were higher for euthanasia (progression), and also showed the only metastases (again progression) in the experiment – none were shown in controls. The abstract is very limited in word length and is supposed to report the major findings most relevant to the results. I see no reason for reporting what they say in the abstract.
SMC: There is no consistent dose-trend - if there were an effect, we would expect to see increases (e.g. in deaths) from 0 to 11 to 22 to 33. In contrast in males, 33 (and C) have the lowest numbers of deaths.
Analyst's comment: This is a dogmatic assumption and assertion of the necessity of linear monotonic dose-response. Endocrine-related effects, such as reported here, are being demonstrated in recent science and discussion papers from the Endocrine Society and elsewhere, to show non-linear non-monotonic dose-response. Seralini et al discuss these findings. The critics fail to acknowledge this literature and the mention in the paper.
SMC: "In females, all treated groups died 2-3 times more than controls, and more rapidly. This difference was visible in 3 male groups fed GMOs." This statement has not been subjected to standard methods of statistical analysis for survival time. The phrase "died 2-3 times more than controls" is based on exceptionally small numbers.
Analyst's comment: I agree it hasn't been tested. The statement on small numbers is misleading as such small numbers of deaths, tumours and pathology can be tested by Pearson chi square if n > 5 or Fishers exact test if n < 5. They should know this. Similarly, the survival times can be tested by appropriate methods.
SMC: The authors suggest a threshold - this rarely occurs in practice. We would expect greatest mortality/ toxicity at the highest dose in a well-designed study. With small numbers as in this study we would expect to see a general trend of mortality increasing with dose.
Analyst's comment: Again, as I noted above, this is a dogmatic assumption and assertion that linear monotonic dose-response is certain to hold. Again, endocrine related effects do not always show this, as can be found in the literature I noted and elsewhere. Seralini et al discuss this and the biochemistry data should show this.
SMC: It seems likely that the numbers in each group are too small for standard methods of statistical analysis to find significant effects on mortality or pathological findings.
Analyst's comment: Again, the sample sizes can be tested by existing methods for significance. I think some significant effects will be found. This criticism is a sweeping statement, an assertion.
SMC: There are virtually no p-values presented. The group sizes are small. This should be interpreted with extreme caution.
Analyst's comment: This is another recycling of the sample size criticism, which is not valid in general. Methods to test exist. There are no p-values because there is no testing done on mortality, tumours or pathology. However, the biochemistry statistical analysis in Fig. 5 contains 99% confidence intervals, which implies a p value.
If the point is that more statistical testing is needed, and what has been done could be better explained, then I can agree that this is the real point.
SMC: It is notable that the figures do not present deaths in the control group in a similar manner (no step graph for controls). This makes it more difficult to compare the other groups with the controls.
Analyst's comment: This is a glaring error on the critics' part and makes me wonder how close and objective the critics' reading of the study really is. Both Figs 1 and 2 show the controls' step graphs in the dotted lines, as explained in the caption of Fig. 1. They are generally to the right, later in time than the treatments, and for mortality are around the control lifespan vertical bar plus or minus SEM grey area also on the right.
SMC: There are many treated groups, and a number of parameters. There is obvious potential for selected reporting, selection of methods etc. In such small groups, with so many parameters this is a big issue.
Analyst's comment: I think this is a cheap statement insulting the integrity of the whole publication process – the journal editors, peer reviewers, and authors. There may be some issues that are valid in terms of volume of results, but again, this is a sweeping statement that lacks any specific examples that might support the criticism. As well, you cannot get everything into one journal paper. The editors won't allow it, and if they do, it becomes a very difficult read.
As well, this comment can be applied to any number of industry-sponsored papers that are selective and use samples of similar size.
SMC: This issue is amplified in the abstract and further in the press release. Strong statements are issued without sufficient backing / explanation.
Analyst's comment: As I said above, the abstract is very limited in word length, so what can authors, any authors of any paper, do? I don't know much about the press release. And I don't see much backing or explanation for many of the comments made by these critics, as I have noted.
SMC: Deaths are compared to the control mean (for males and females).
Analyst's comment: I don't see any calculation of the control death numbers mean (which are binary), only mean lifespans.
SMC: Due to the distribution of deaths (most deaths occur in old age), this is almost bound to exclude the large majority of deaths in the control groups.
Analyst's comment: Is this comment directed only at controls? Yes, most deaths for controls occur in old age (lucky rats, right?), unlike the step graphs of the treatments, which appear to occur early. And since the lifespan of the SD rat is about the 2 year timeline of the study, I cannot really make sense of the comment. The experiment is a comparison of controls and treatments of the same rat, so wouldn't the same natural lifespan influence both groups? The real point of the experiment results on this count is that the treatment rats tend to die earlier, and in greater numbers, and show greater pathologies. How can this be rigged to exclude deaths among the controls?
SMC: The 2 or 3 deaths in the control group is determined by their methods, but is inappropriately presented as a true observation.
Analyst's comment: Again, this continues the sentences above that don't make sense to me. As well, another criticism of Seralini et al is that the Sprague-Dawley rat is susceptible to disease and tumours, so would not this enter the matter as a sensitive control?
How is it determined by the methods? It is an observation of a result of the experiment. How is it misrepresented?
I would reply that the 90-day study presents a similar issue for determining deaths and other pathologies, but in the controls.