Dialling back ‘impact’ claims: researchers should not be compelled to make policy claims based on single studies

Background

Researchers are increasingly expected to draw policy implications from their research, yet this can be distracting or misleading when describing single studies. Rather than helping ensure that research benefits society, it may distort research and evidence-based policy. This is propelled by incentives such as career structures increasingly favoring evidence of ‘impact’ and a need to appeal to competitive publication and funding decisions. We discuss this issue, use an example from the health inequality literature to highlight the complications of drawing policy conclusions, and consider options to improve both research and evidence-based policy making.

The incentive structures in science are problematic.^1,2 Fraud cases are numerous, results are often overhyped and major research papers across many fields are not reproducible.^1,2 In some ways the problems appear to be getting worse. Recent work has found the use of ‘hype’ or positive language has increased across time in scientific articles^3,4 and funded grant applications.⁵

There are also increasing financial and career incentives to produce research judged to lead to broader societal (policy or political) impact. Take, for example, the UK's impact agenda—substantial financial returns are tied to ‘impact’ (25% of university core funding⁶), defined as the ‘demonstrable contribution that excellent research makes to society and the economy’.⁷ This agenda ostensibly rewards the broader contribution of research, unlike narrower indicators of research productivity (e.g. citation metrics), yet there are increasing concerns about its downsides.⁸ Unsurprisingly, given the large incentive, hype is common in impact submissions,⁹ and universities spend significant sums on impact generation (at the expense of research).^9,10 There is increasing awareness of the importance of impact for scientific career progression, including awards provided by research funders. Editors of journals operate in an increasingly competitive landscape, and authors must make their articles appealing to publish in ‘top’ outlets. Papers or grant proposals which purport to advance policy may be preferred to those which merely ‘add’ to the scientific literature¹¹; outputs from such research may garner more media and policy interest, and ultimately might lead to ‘demonstrable’ impact (e.g. being cited in a policy report, attracting media attention or being a claimable cause of policy change).

We recognize the importance of policy relevance of research (the questions we ask and try to answer) but we are concerned that misaligned incentives are leading to researchers increasingly drawing—or at least being compelled to draw—unfounded policy conclusions based on single studies. This is despite an increasing understanding of causality (the ‘causal revolution’)^12,13 and previous calls to promote evidence synthesis and triangulation rather than individual studies.^14,15 Our concerns are far from new. For example, Rothman and Poole in 1985 warned that ‘the conduct of science to achieve political ends will corrupt both endeavours’.¹⁶ As of 1993, Epidemiology explicitly requested that policy implications are contained in commentaries and not in original research articles; illustrating the wider draw of ‘impact’; however, more recently established journals which encourage the opposite have considerably higher impact factors (e.g. Lancet Public Health—its impact factor is reportedly almost 10 times that of Epidemiology).

Researchers are incentivized to follow an alluring structure: describe results of a single study, briefly outline its limitations, and then go on to make broad policy implications. Such pronouncements often appear in the Discussion (Implications subsection) and sometimes in the main abstract. This is to be expected: we are influenced by upstream processes and incentive structures. We have not cited examples, as it seems unfair to pick out individual authors; we are all exposed to this force, as an audit of our own previous publications would demonstrate.

In the context of work which tackles sensitive topics—such as childhood obesity, health inequalities, youth mental health or social mobility—the lack of political will to address key policy issues is an additional driver of this trend. It leads to frustration among authors, reviewers, editors and grant committees with merely describing associations, encouraging more ambitious policy or political conclusions. However, when articles make unfounded policy conclusions based on single studies, they risk undermining trust in the discipline and science as a whole.

Below, we describe how even strong associations between socioeconomic factors and health have a surprising number of complications which make drawing straightforward policy conclusions challenging. This leads us to suggest how we as authors should report on, and as reviewers should provide feedback on, policy claims made in future.

Policy implications of associations between socioeconomic factors and health

The study of how social disadvantage associates with worse health is an important topic across many disciplines—notably (social) epidemiology and public health, medicine, demography and economics. Work in this field is often motivated by a noble desire to raise awareness of and ameliorate a particularly pervasive form of injustice: those with fewer resources in society are at higher risk of ill health and premature death. These links have been repeatedly documented for decades,^17,18 motivating policy nationally and transnationally.

A substantial share of this research is necessarily descriptive in nature, since randomized controlled trials, as used in medicine, are often impractical or unethical. However, researchers reporting descriptive work in this area often include (or are compelled to include) broad policy implications. This is in contrast to the evidence-based medicine approach, in which clinical recommendations result from systematic reviews and meta-analyses; single studies [e.g. randomized controlled trials (RCTs) of drugs] often focus solely on the study description rather than engaging in broad policy discussion of what the results might mean for future policy. The policy implications of links between social disadvantage and health are complex—they depend on causal processes we have imperfect evidence about and which are likely context-specific. They also require difficult policy considerations more broadly. For example, consider the following illustrative inference of only 42 words, as is often necessary given tight word limits:

‘We reported a strong social gradient in health; those with lowest education attainment had a 3-fold higher risk of cardiovascular disease compared with those of highest education. Policies to improve education attainment could reduce health inequalities and benefit overall population health in future’.

Complications implicit in this kind of statement but often not discussed are plentiful:

• There is a causal link between education and health—a suggestion supported in traditional observational studies but with mixed findings in the (quasi-)experimental literature.¹⁹

• The beneficial effects of education will be translated when intervened on in the future (i.e. the consistency assumption²⁰ is met and effects generalize across different contexts). Policy initiatives to increase the length of education may have effects that differ by context (i.e. time and place). In one example in which longer schooling was not associated with improved educational outcomes, greater education was linked with worse health outcomes for those from lower social classes.²¹

• The exposure of interest (education) can be translated into a policy intervention: it is possible to intervene on the educational gradient. It remains unclear which dimension of education actually matters for health. As noted above, changes in the duration (quantity) of schooling have produced mixed effects on health, and the literature on changes in the quality of education (e.g. class sizes, curriculum or tracking by ability) is in its infancy.²²

• Policies to improve educational attainment will not have other unexpected negative consequences to public health (e.g. mental health) or to other societal (e.g. economic) outcomes.

• The way the effect of interest is presented is key. In our example, only the relative risk was presented; knowledge of the absolute risk of the outcome in each group is typically needed to inform policy decisions.

Further, a broader discussion of the different policy options is not included. Any policy decision necessarily has opportunity costs and trade-offs: for example, the choice between investing more in education or health care or social care. Assessing likely costs and benefits of policies are complex analyses that must be done carefully. This key issue which faces those making policy decisions is not explicitly addressed in the above statement. This is true even when decisions have to be made quickly on limited evidence (as happened during COVID, for example).

A more cautious short-form alternative could be provided, which may well be considered uninteresting by those seeking impact from individual studies. For example: ‘Assuming links are replicable and causal, translate when intervened on and do not have other negative consequences, increasing educational attainment could be one policy option to reduce health inequalities and improve overall population health. Further research is required to test these assumptions and to conduct cost-benefit analyses of the multiple policy options available’.

Note, these issues are also relevant for systematic reviews: descriptive studies do not become any less descriptive simply because they are combined. They also largely apply to quasi-experimental or experimental studies: even with less potential for confounding, policy conclusions are not always clear-cut. Results may not transfer between settings, there may be unaccounted for costs or benefits, and the overall gains from the policy may not surpass its opportunity costs.

Suggestions for authors, reviewers, and funders

Standard research articles in epidemiology/public health are concise, with word counts of 3000–3500 words. Researchers are often in the unfortunate position of being expected to draw policy implications from their research at extreme brevity. In the internet era, concerns regarding word limits will surely be less relevant; if authors want to include policy discussion, editors should consider letting them have more space to do so comprehensively. If they are included, they should be written carefully, and humbly,¹¹ drawing on evidence from multiple sources. If authors decide not to include such discussion and focus instead on other important aspects of their work, this should not be penalized by editors, reviewers, funding panels or academic career structures. This applies to authors of both original research articles and review articles.

Researchers and reviewers often lack training or expertise in policy analysis or cost-benefit analysis; future students could be trained in such work, so that researchers working on all types of study designs: (i) are aware of the assumptions required to infer causality from their work; (ii) understand how their research could contribute to evidence-based policy; and (iii) can identify policies which could be appropriately analysed with available data.

Researchers have increasingly constrained bandwidths, and it is debatable whether a brief discussion is helpful for many (or indeed most) single studies; if policy discussions are not included, other valuable aspects can be expanded (e.g. more detailed introductions, methods, results or discussion sections; or triangulating their findings across multiple datasets), or new resources provided (e.g. providing analytical code which only ∼2% of researchers do²³). Policy discussion could instead be contained within separate pieces of work, or it could be accompanied separately alongside original research articles; the latter is the approach taken in social science genetics research, with policy-relevant frequently asked questions (FAQs) appended separately to avoid misinterpretation of their results.²⁴ This may be optimal, given the challenging and specialized nature of contemporary research; it may yield more robust, reproducible science and improve evidence-based policy.

Policy recommendations should generally be based on literatures, not individual papers, and on the myriad of considerations noted above that policy decisions require. Researchers are often motivated by a noble desire to improve society and influence policy; yet ironically, policy discussion articles are typically less readily publishable or valued than original research articles. If academic research aims to affect policy, separate detailed policy discussion pieces or reviews should be valued by editors, funders and academic career structures. Policy discussion pieces should be valued as separate pieces of work where they can consider causality, generalizability and policy trade-offs in sufficient depth. In contrast, researchers should be less compelled to make policy claims based on single studies. All types of study, whether observational or experimental, have value in this context, even if individually they cannot ordinarily lead to broader impact.

Author contributions

D.B. wrote the first draft; all authors contributed to discussions and revisions and approved the final version.

Funding

D.B. and L.W. are supported by the Economic and Social Research Council (grant number ES/M001660/1); D.B., L.W. and N.M.D. by the Medical Research Council (MR/V002147/1). E.C. is supported by a UKRI Pioneer grant (EP/Y010345/1). N.M.D. is supported by a Norwegian Research Council Grant number 295989.

Conflict of interest

None declared.