Their analysis incorporated 42 studies and is unique because data were supplemented with weather observations used to quantify moisture and temperature stress. Results from the general linear mixed effects model employed indicated that CA yields improve with increasing moisture and heat stress although these effects are partially controlled by soil texture. Previous studies also suggested that N fertilization can offset lower CA yields . Steward et al. conversely provided new evidence that increasing N rates in CA does not improve maize yield under drought. And while previous meta analyses suggested that rotation improves CA yield , Steward et al. found little supporting evidence, although diverse rotations were found to reduce yield variability under heat stress. Last, Knapp and van der Heijden also framed their paper as a ‘global meta-analysis” in terms of the need to match growing population and food demand with sustainable productivity increases. They highlighted that temporal yield stability under CA remains poorly understood. A total of 2453 comparisons from trials at least 4 years in length were made by re-analysing data extracted from Pittelkow et al. , 60% of which came from North America. Absolute and relative stability YRRs were calculated as the ratio of experimental to treatment standard deviations or coefficients of variation across observational years, respectively. They found that temporal yield stability under NT does not differ significantly from conventional tillage,blueberries in containers and that the transition to NT does not affect yield stability.

Knapp and van der Heijden also discussed the limitations of experimental plot-scale measurements of yield stability relative to farm-scale measurements with multiple crops and crop rotations. In order to improve yield stability at this scale, they suggested ways that farmers could cultivate different crops in different fields to overcome poor performance of particular species in particular fields. They also suggested that use of species and genotype mixtures to reduce risks of crop failure.By applying powerful statistical analyses to large datasets constructed using primary literature, meta-analysis is intended to arrive at unifying conclusions and provide clarity in research . Use of meta-analysis may also be described as part of the logical-positivist paradigm, in which researchers justify and frame their work in terms of the primacy of hypothesis-driven and empirical inquiry. However, researchers’ paradigms can also be influenced by their politicized worldviews, beliefs, and perceptions of reality, in turn affecting scientific framing . Our review, which recognizes the socially embedded nature of agricultural research , suggests that the variable application of methods and contestation over the framing and justifications given for research questions can undermine the purpose of meta-analysis to provide definitive conclusions. Do the large sizes of databases and reportedly comprehensive analyses conducted with meta-analyses matter? Our case studies of OA and CA meta-analyses indicate that meta-analysis appears to fuel rather than diminish controversy. This is particularly the case for meta-analyses framed as contributing evidence to discourse that productivity increases are requisite for feeding a global population and assuring food security in smallholder agriculture.

We review these issues by discussing three considerations for how both scientists conducting meta-analysis and readers of scientific literature can more carefully evaluate meta-analytical evidence, particularly when applied in the context of development-oriented agronomy.The field of science and technology studies has long acknowledged the problematic but necessary role of experiments in advancing knowledge. Experiments are a social construct intended as a simplified version of reality . When designing cropping systems trials, agronomists make choices regarding the grouping and organization of a range of component crop management practices into standardized categories that can be mechanistically implemented across replicates. Yet as shown in our case studies, scientists encounter tension between the generation of experimental evidence and the need to justify their studies in terms of research investment and/or development relevance, and development impact . Our review also highlights an additional weakness of meta-analysis when applied to agronomy. While experimental standardization permits replication and statistical inference, this process can actually decouple chosen management practices from the contextual realities of the farming systems are meant to represent. This limits the degree to which agronomists can responsibly extrapolate and discuss the implications of field trial results. Meta-analyses in agronomy appear to amplify this problem.

Researchers conducting meta-analyses make additional choices regarding what treatment combinations and experimental procedures in primary studies they consider appropriate and admissible to their databases. Yet when databases are built on treatments that are debatably inappropriate, or that are highly decontextualized representations of farming systems realities, researchers extrapolating their results may inadvertently reduce the value of their studies to provide relevant and unifying conclusions. The OA case study particularly highlights how differing paradigm and opinions regarding what does or does not constitute an appropriate treatment may render debates difficult to resolve, regardless of statistical power accrued by using meta analysis. While screening literature for their systematic review, Badgley et al. for example grouped ‘…farming practices that may be called agroecological, sustainable, or ecological; utilize natural nutrient-cycling processes; exclude or rarely use synthetic pesticides; and sustain or regenerate soil quality… [and] include non-certified organic’ as ‘organic’ in their analysis. This definition, which is arguably broader than most organic certification standards, resulted in a number of studies in which synthetic fertilizer had been applied being counted as OA. Subsequent OA meta-analyses by Seufert et al. and de Ponti et al. therefore applied formal organic product certification standards as the baseline criteria for literature and data selection. Ponisio et al. later strongly framed their analysis in terms of the ‘imperative that we adopt sustainable and resilient agricultural practices as soon as possible’. They equated OA with generally better performance than conventional practices when sustainability indicators were considered, and therefore justified their research as an investigation into how crop diversification affects OA performance. Primary data were therefore collected from databases using Boolean searches for the terms ‘organic’ and ‘ecological’ with ‘agriculture’, ‘production’, ‘cropping’ and ‘yield’, as well as ‘compare’. They however did not provide clear definition for the specific management practices that constituted ‘ecological’ practices. This distinction is important because ‘ecological agriculture’ is generally broader than OA,planting blueberries in pots and may make strategic and targeted use of synthetic inputs and may or may not conform to organic standards . With the exception of Van den Putte et al. , whose definition of what constitutes reduced and conventional tillage was not fully specified, the criteria used to define different configurations of CA principles in the meta-analyses reviewed in this paper tended to be more specific . Such differences highlight that meta-analyses in agronomy are socially and politically situated, and that this may affect the definition of cropping system comparisons and literature selection. To address these problems, Cassman proposed that cropping systems comparisons should only be considered where ‘best management practices’ are clearly specified and employed for each system studied. Yet defining what constitutes ‘best management’ is already a contested issue, and may be complicated by differing values and agricultural research paradigms . Indicators of ‘best management’ and cropping systems performance may also trade-off with each other.

Selection of appropriate criteria for ‘best management’ practices therefore entails a degree of subjectivity. Questions of disciplinary authority and legitimacy may also arise over who is qualified to determine what constitutes ‘best management practices’. This appears to be particularly relevant in debates where development oriented agronomy plays a role in research framing and justification, including OA and CA, among other topics such as the System of Rice Intensification . An example of these problems is provided by the CA case study. A widely recognized advantage of RT is to forgo time- and energy-consuming repetitive plowing. Tillage can also delay sowing and crop establishment, which may lower yield potential . Under these circumstances, RT and early sowing could be considered a ‘best management practice’. Yet in order to improve comparability of treatments , more than half of the meta-analyses reviewed in this paper included data from studies in which CA and conventional treatments were established on the same date . This observation underscores how the choices that scientists make when designing experiments and meta-analyses to isolate treatment effects may result in a decoupling from farmer realities. Keil et al. , for example, argued that in the context of eastern India, Pittelkow et al.’s conclusion that NT reduces wheat yield is invalid because farmers often utilize NT practices to advance wheat sowing dates. In addition to production cost reductions, this permits the crop to escape from yield-reducing late-season heat stress. Keil et al. illustrated this point with farm survey data, backing earlier observations by Erenstein and Laxmi in north eastern India where NT has been adopted by farmers on over 1.5 million ha, resulting in an estimated 5–7% increase in wheat yields.These observations – which appear to be amplified in meta-analyses of OA and CA that use large datasets and claim increasingly comprehensive research results – represent the inherent tension in empirical studies that arises when scientists define treatments and interpret the implications of their results outside the experimental setting . A first step in addressing this issue is to formally recognize that cropping systems may not be as simple in reality as compared to experimental settings. In other words, crop management follows a wide and variable range of practices when implemented by farmers as compared to experimental agronomists . Rather than relying on perhaps artificial groupings and diametrically opposed comparisons of OA or CA versus conventional management, research could also consider the importance of gradients in crop management. A preliminary example is the meta-analysis conducted by Hossard et al. that compared a range of LEI systems to organic and conventional agriculture. Another perhaps more important and general suggestion is to more conservatively interpret and extrapolate the implications of experimental and meta-analytical research results.As described previously, the ways in which cropping systems are defined and studies are screened for meta-analysis can influence the interpretation of research results. The ways in which scientists conceive of and frame their research questions is of similar significance. Conceptual frameworks are important in the organization of human experience, perception, and understanding, including scientific paradigm . The boundaries imposed by a conceptual framework help to define what information may or may not be considered valid in the evaluation of research evidence to inform decision making . Most of the meta-analytical studies of OA and CA reviewed in this paper justify their work as contributing to development-oriented agronomic goals, e.g., meeting global food production, food security and environmental sustainability goals. In comparison to localized and context-specific research results, research that is framed as answering questions of broad global significance can arguably increase scientists’ chances of publication and citation. Where h-indices are highly rewarded in evaluating a researchers’ accomplishments, concerns have also emerged that meta-analysis can distort scientific integrity . Leeuwis et al. and Andersson and Sumberg also point out that framing research in terms of global development impact is important for securing funding. Scientists discussing the implications meta-analytical results beyond the plot scale however face a number of important challenges. For example, Corbeels et al. and Rusinamhodzi both examined maize yields under CA in sub-Saharan Africa. Both studies are clear examples of meta-analysis framed within the political economy of development-funded agricultural research . Having a clear and practical development orientation – where and by which farmers can yields can be increased through CA? – Corbeels et al. and Rusinamhodzi et al. compared NT and treatments inclusive of all three CA principles to conventional tillage practices in which residues were removed from fields. Lundy et al. and Pittelkow et al. conversely reframed the debate that had emerged over the benefits of CA in sub-Saharan Africa in terms of meeting global food needs, with emphasis on agricultural resource management and productivity in Africa and South Asia. Lundy et al. and Pittelkow et al. however did not include treatments in which full tillage was practiced with residue removal. Rather, they applied a control treatment in which residues were incorporated into the soil during tillage. This was done in order to isolate yield responses to tillage alone in comparison to NT with or without residues. Although subtle, such standardization illustrates an important point regarding the choices researchers make in defining crop management practices for treatments and their implications when extrapolating and discussing meta-analytical results.