A heavily promoted meta-analysis claiming to demonstrate the benefits of GM crops, especially in developing countries, does no such thing, writes Prof Jack Heinemann
EXCERPT: It is very difficult to get solid indicators of the actual contribution GM cropping makes in places in which it is being used. This is why I question the unequivocal statements made by some scientists about benefits. Their conclusions appear to be uncoupled from the strength of the data. [Find the three charts at the original link.]
Correlation is not causation
Jack Heinemann
Rightbiotech, 27 Nov 2014
http://rightbiotech.tumblr.com/post/103665842150/correlation-is-not-causation
* A heavily promoted meta-analysis claiming to demonstrate the benefits of GM crops especially in developing countries does no such thing.
* It fails to separate from the GM trait the effects of confounding variables that are known to dramatically affect all measured parameters.
* Commentators are presenting misleading interpretations.
A new meta-analysis of 147 studies is cited as evidence that GM crops have made substantial yield and profit increases and reduced pesticide use (Klumper and Qaim, 2014). However, it fails to differentiate between, on the one hand, differences in farmer income, ability to irrigate, and access to support—and importantly access to elite germplasm — and, on the other hand, the contribution of the GM trait. In other words, the meta-analysis confuses correlation with causation.
Those quick to overstate the power of this study seemingly ignore that it is prone to distortion from environmental and temporal heterogeneity because it is a collection of mostly short-term observations lacking geographic continuity, disconnected in time and without uniform standards of measurement. This it has in common with some other meta-analyses that have been used to make similar claims about the certainty of benefits from GM cropping systems.
Findings at a glance:*
* The results cannot be generalized to the global experience. Almost all studies come from just three countries: South Africa, the US and India (Figure 1).
* Almost all studies were under three years long, with ~60% of studies just one season (Figure 2).
* Almost all time periods measured were prior to the rise of resistant weeds and pests, making measured benefits a short-term phenomenon.
* Only three crops were examined: cotton, maize, and soybean. There was no coverage of canola, sugar beet, alfalfa, or papaya.
* Nearly 80% of studies are on Bt (insecticidal) crops, despite the majority of GM crops for the majority of time being HT (herbicide tolerant) (Figure 3).
* Over 50% of the studies are just Bt cotton. Only 14% of the studies were of HT soybean despite this combination of crop and trait being the longest in commercialization. Moreover, these minority studies are split over multiple countries, making them anecdotal.
* Any of six different variables qualified a study for inclusion, reducing the number of comparisons on some variables to well below a sample size of 147. The mix of methodologies in the underlying research papers included is so varied that the collection has almost no power of true replication.
Correlation not causation
Does the meta-analysis demonstrate that GM crops are in general superior to non-GM counterparts for yield, pesticide use and profitability? No. At best, taking the meta-analysis at uncritical face value, it presents a correlation between GM crops and some measured benefits. It does not provide evidence that GM-traited crops are responsible for the benefits. When one looks deeper than face value, the correlation itself is weak.
Many of the underlying studies that I examined* were based on farmer surveys (rather than actual measurements). Not only do farmer surveys weaken the strength of the overall conclusions, but some of the underlying studies used in this meta-analysis have been subject to significant methodological criticism (Glover, 2010c). Studies 57 and 59 (and the research groups responsible for studies 45-46, 55-59 and 100) were described this way by Glover (2010c) —
"Work by other researchers has shown that these are not simply mischievous speculations. Fundamental questions have been raised about the Pray–Huang group’s conclusions, with regard to both the finding of reduced pesticide use and the attribution of such an effect to the adoption of Bt-cotton technology per se."
Many of the underlying studies were surveys of outputs from farmers, some of whom grew GM crops and some who did not. For example, at least 14 studies (10-17 and 52-55 on cotton and 41-42 on maize) appear to have assembled data from an area where some farmers were growing GM and other farmers conventional versions of the crop. This is nearly a third of the studies I examined. The meta-analysis should have, but did not, take into account any systematic differences between farmers who adopt or do not adopt a particular variety to ensure that differences are due to the GM trait and not something else.
Studies that compare farmers who adopted GM crops with those who may have been growing saved seed or other commercial varieties are introducing significant variables, e.g., different wealth or other characteristics of the farmer or farm. GM crops are also elite varieties, products of the latest breeding that has nothing to do with the GM trait. Companies selling high-cost GM seeds do not undermine their market by breeding the GM-based trait into poorly performing germplasm. Those farmers adopting GM are also adopting both the latest germplasm and a management program designed by the seed seller. GM seed companies have programs that finance small scale, early adopting farmers (Glover, 2010a). Other farmers are working with germplasm that is not related to the GM versions and they do not have access to the same level of outside support. Studies collected for the meta-analysis are biased toward early adopters.
"The problem is key because almost all studies have focused on the years immediately following the introduction of Bt cotton, when yield differences mainly reflect the agricultural prowess of a biased group of early adopters (and also reflect how this group happened to fare their first time trying a new technology). Crost et al. found that in “cross-sectional analysis of the type used in most of the previous studies on Bt cotton, more than half of the observed yield effects would be due to self-selection effects”… A related problem is bias in cultivation practices: prior to the institution of price caps in some [Indian] states in 2006, Bt seeds cost four times as much as conventional seeds, and would have been planted in the fields with best irrigation and then benefited from unusual care and expense." —(Stone, 2011)
In effect, these studies are a measure of the impact of breeding and socio-economic factors, not GM. The GM trait may correlate with greater yield or profitability, but that correlation is not a demonstration that GM is the cause of these benefits.
Study choices
The authors of the meta-analysis were candid in their selection procedures. To be included, a study had to provide some original data (and not be just another meta-analysis). They also included multiple studies with duplicated data—
"In some cases, the same results were reported in different publications…several publications involve more than one impact observation, even for a single outcome variable, for instance when reporting results for different geographical regions or derived with different methods (e.g., comparison of mean outcomes of GM and non-GM crops plus regression model estimates). In those cases, all observations were included."
The addition of an entirely new paper to add one small data contribution inflates the power of the meta-analysis. For example, studies 34 and 35 comprise a conference presentation and later publication of the same dataset. Plagiarism software found a 71% identity between the publications, and most of the differences were just cosmetic. There may have been a unique data point between the two, but this small difference added another entire paper to the list.
Meanwhile, while entire studies were included for the sake of a small additional piece of data, one of the largest studies from a developed country was excluded from the meta-analysis. This study, on one of the longest used GM crops, examined both yield and profitability characteristics of GM cotton over 4 years in the same place (the US State Georgia, see Jost et al., 2008). The study is over a long period of time compared to most of those used in the meta-analysis, and was conducted over consecutive years in a country where all farmers are able to access the latest germplasm. That study did not, however, find that GM crops were superior to conventional.
In fact, the Jost et al. study found that GM-traited seeds eroded farmer profits because of their high cost. Worst performing was the HT GM cotton. In this regard, it is noteworthy that HT studies are a minority in the meta-analysis, despite HT being the most common GM trait and the longest in commercial use. Jost et al. concluded —
"Collectively, these results indicated that profitability was most closely associated with yield and not with technology."
That is the key. If farmers using obsolete germplasm are compared to those using the latest elite varieties (packaged with a GM trait) and provided with support to optimize management, the latter will be found to have greater yields. Nothing about this proves the benefit of GM. Moreover, those with greatest yield make more money. Again, the outcome is correlated with yield and GM. But only yield is causative and this is determined by factors other than the GM trait.
Contrasts
The meta-analysis seems more suited to providing a snapshot of two agro-ecosystems, India and South Africa, comparing production of just Bt cotton (and perhaps maize). Drilling down into the studies specific to those agro-ecosystems could be useful, rather than conflating them with one-off measurements from many other countries and mixing together on farm studies and field tests as was the case with this meta-analysis.
Confusingly for me, the authors say that they excluded studies on sugar beets, papaya, and canola (oilseed rape) for precisely the reason that there were, in their view, too few studies outside of a small number of countries. But then to include a few studies on HT soybean mainly in the US (with 3 studies in nearby Canada and only one other study from Romania) seems inconsistent.
The meta-analysis at its most authoritative is not a validation of GM cotton, soybean and maize superiority, much less a general endorsement of all the kinds of crops that have been genetically modified. Realistically, the meta-analysis is built on some shaky underlying information and lack of detail on breakdowns where there is greatest replication power (Glover, 2010c; Stone, 2011).
The authors also ignored our analysis of performance of crops grown at scale in countries that are matched for hemisphere, access to elite germplasm, mechanization and farmer support for the longest ever measured period of over 50 years (Heinemann et al., 2014b). Our study did not compare matched GM/conventional crops grown side-by-side. However, neither did most of the studies in the meta-analysis as I indicated above. Study 30 of the meta-analysis was explicit about that —
"It is important to emphasize that this is only a cross-sectional survey. It does not represent a side-by-side comparison of GMO and non-GMO crops. It represents a picture of what Iowa farmers experienced."
Our long-term study was criticized for this very thing, but some of the same voices are silent about the issue when it is used to produce evidence of GM crop benefit.
Moreover, our long-term study assembled measured farm outputs, so it should have met the criteria for inclusion. However, when we examine the use of GM crops in North America and elite equivalents in Western Europe over the short period of GM adoption, we find yield stagnation or decline in North America (Heinemann et al., 2014a).
Importantly, we also found the same yield penalties apply to wheat grown in the US but not Western Europe. Wheat is a major non-GM crop in the US and Western Europe. This is further evidence that one needs to consider the entire biotechnology, management, and innovation package under which agriculture is conducted in order to determine which tool in the toolbox will be truly useful.
Lessons
It is very difficult to get solid indicators of the actual contribution GM cropping makes in places in which it is being used. This is why I question the unequivocal statements made by some scientists about benefits. Their conclusions appear to be uncoupled from the strength of the data.
All studies have their weakness. It would have been useful if we could have had massive side-by-side comparisons of GM and non-GM plants and management systems for our 50-year study. That kind of data at that scale does not exist. But we did balance as much as possible the larger variables of wealth, education and biotechnology. Likewise, it would have been useful if the latest meta-analysis would have included all relevant studies, more crops and only side-by-side comparisons, rather than mixing data in all ways possible.
GM plants are not science: they are products made by scientists. They may or may not do what their developers hope for them. Likewise, alternatives to these products may also fail to live up to expectation. Therefore, we need proper holistic evaluations of the tools in order to inform policy on the future of agriculture.
"Public policy requires, and small farmers need, a more rigorous and dispassionate evaluation of the pros, cons and opportunity costs of GM technology and especially a more careful analysis of how it can be designed and made available in forms that could enable it to be authentically ‘pro- poor’." — (Glover, 2010b)
Skepticism about the performance and value of one or more GM products is not a rejection of science or scientists, any more than switching power companies is a rejection of electricity. Those claiming that these products are science are failing to distinguish between science and technology. Perhaps this is not unexpected given that the three words are now spoken as if they were one word. They never have been and are not now, and it would serve us well to remember that.
* How my analysis worked
I looked at ~one-third of the studies, a significant subsample. I applied no rule, simply taking the first 49 of 147 studies (Table S1 of original paper). In the end, I had to consider the first 55 studies because I either could not find or was unable to access relevant details of studies 5, 7, 18, 38, 43 and 48. Only 2 traits are covered: insecticidal Bt and herbicide tolerance (HT).
One study had data from two countries. From this subsample and a scan of the titles of other studies, I estimate that the geographical diversity is largely saturated in the first 55 studies with the total number of countries from which data was obtained for at least 1 study estimated at being ≤15.
References
Glover, D. (2010a). The corporate shaping of GM crops as a technology for the poor. J Peasant Stud 37, 67-90.
Glover, D. (2010b). Exploring the Resilience of Bt Cotton’s ‘Pro-Poor Success Story’. Develop Change 41, 955-981.
Glover, D. (2010c). Is Bt cotton a pro-poor technology. J Agrarian Change 10, 482-509.
Heinemann, J.A., Massaro, M., Coray, D.S., and Agapito-Tenfen, S.Z. (2014a). Reply to comment on sustainability and innovation in staple crop production in the US Midwest. Int J Ag Sustain 12, 387-390.
Heinemann, J.A., Massaro, M., Coray, D.S., Agapito-Tenfen, S.Z., and Wen, J.D. (2014b). Sustainability and innovation in staple crop production in the US Midwest. Int J Ag Sustain 12, 71-88.
Jost, P., Shurley, D., Culpepper, S., Roberts, P., Nichols, R., Reeves, J., and Anthony, S. (2008). Economic comparison of transgenic and nontransgenic cotton production systems in Georgia. Agron J 100, 42–51.
Klumper, W., and Qaim, M. (2014). A meta-analysis of the impacts of genetically modified crops. PLoS ONE 9, e111629.
Stone, G.D. (2011). Field versus farm in Warangal: Bt cotton, higher yields, and larger questions. World Develop 39, 387-398.