Introduction
Carefully developed recommendations for conducting studies of programme effectiveness have provided an extremely useful framework for researchers and articulated the different components required to answer not only the question of effectiveness (does it work?), but also the equally important questions of how, why and for whom was the programme effective.1 However, the continued emphasis on the use of the term ‘complex’ in describing the intervention itself or the system within which it is contextualized can detract from our ability to focus on the strategies required to better understand the strengths or limitations of the evidence base for decision-making.
Social and public health interventions operate in a context that demands explicit recognition of politics, service systems, funding flows and shortages, staff competencies and multi-strategic approaches. For those immersed in the complicated political and scientific acrobatics of coordinating studies of programme implementation and evaluation, any real or perceived misalignment in the connection between research effectiveness trials versus policy relevant implementation knowledge must be overcome. Devising relatively simple approaches to understand complexity can assist in making complexity more manageable so that meaningful answers to important policy and practice questions can emerge.
In this paper we argue that if reviews of intervention evidence are to be useful to decision-makers at all, contextual and implementation information is an essential, non-negotiable component of the review process. We highlight steps evidence review authors can take to capture and interpret this information. With relatively small changes or additions to the evidence review process, practical, meaningful and rigorous public health evidence can be generated.
Practical steps for embracing complexity and ensuring policy relevance
Engaging stakeholders to ensure utility of the scope
Within the context of rigorous and transparent review methodology, there is a strong need for the scope of public health evidence reviews to be shaped by end users: stakeholders who are uniquely placed to have a deep understanding of the nuances inherent in the research and policy environment context. Having this perspective early on in the development of the protocol for the evidence review will go a long way to ensuring relevance of the evidence captured and reviewed in the process.
For example, in an upcoming Cochrane public health review funded by 3ie,2 end users have been involved in a review advisory group (RAG) to assist in defining slum upgrading interventions: a range of strategies from installing basic infrastructure, home improvements, securing land tenure, improving access to health and basic services through to integrating physical, social, economic, organizational and environmental improvements in slums. Given that the range of strategies is too broad to be assessed in one review, identifying the core component of all programmes to be included was essential in narrowing the scope. In this case, physical environment interventions (with or without other approaches) usually represent the most basic level of slum upgrading, thus enabling the review findings to be relevant to a broad range of end users. The RAG served an important role in guiding this decision.
Understanding the pathways operating and theoretical underpinnings of the evidence
Logic models can be used for many purposes including designing, implementing and evaluating interventions. They can also be used in a systematic review to detail the process of how an intervention might be expected to achieve particular outcomes and what aspects of the process are under review. Coming to a shared understanding of how the intervention is likely to improve public health can be difficult when stakeholders are involved from different disciplines and sectors, however achieving this ensures that the outcomes will be useful and have wide relevance. Mapping pathways and presenting logic models also assists the reader of the review by ‘setting the scene’ and explicitly articulating the theoretical assumptions of the review authors. This logical model can also guide the presentation of results and synthesis of findings by the review authors, potentially making the review more readable.
The inclusion of logic models in systematic reviews of complex interventions has been described in detail elsewhere.3 We echo these recommendations and suggest that logic models are a useful and important way of demystifying a complex process and making clear the ‘logic’ of the review parameters, in ways that are immediately (and visually) explicit to the reader of the review.
Capturing programme implementation through the data extraction process
Implementation and effectiveness are inextricably linked. For example, an intervention may not have achieved its desired outcomes, and therefore deemed ineffective, however on closer examination it may become apparent that the programme was not implemented as planned. By collecting and examining information about implementation, researchers and subsequently review authors can gain insights into both why an intervention may not have shown effect (for instance, the intervention may have not reached the audience it intended to or to the extent that it would need to in order to be effective), and which components of an intervention may have been most effective. Further, by enabling the reporting of the cost and resources required for implementation, the review has much more utility for practitioners and decision-makers. To provide this information, evidence reviews reporting on the effectiveness of complex interventions can, at a minimum, report information which captures the process of what was implemented in each of the primary intervention studies.
The driving questions therefore relate to how an intervention might work, for whom, under what conditions and at what cost. Given the substantial investment of time required to extract data related to effectiveness, it may seem like a daunting and unnecessary task to answer these additional questions. However, we contend that the available evidence can be probed by the addition of a small number of questions to the data extraction form to provide the information of interest. This extracted data can then be included in the table of the characteristics of included studies. The additional questions for data extraction may include the following:
What is the organizational or political context in which the intervention has been delivered?
What (if any) were the partnerships involved?
What was the theoretical basis for the intervention?
Who initiated and developed the intervention?
How was the intervention delivered? Were strategies included to reach disadvantaged community groups?
What resources (funding, staff hours, materials, equipment etc) were required for implementation?
Were process data collected or implementation measured? If yes, what was the level of implementation achieved?
Was an economic evaluation conducted? If so, was the programme cost-effective?
How much additional time is required to collect these data?
Based on our experience piloting these questions in the recently updated childhood obesity prevention review,4 we estimate that this information can be extracted from included studies by investing an additional 30–40 min per paper. If additional papers have to be retrieved (because the relevant information is published separately) there will be additional investments of time required. For a review of 10 included studies, this information would only take one person about 1 day to extract. If however, this information is going to be retrieved post hoc, to assist in the synthesis and translation of findings, then a much larger investment of time will be required, as the studies will essentially need to be reassessed and findings reinterpreted.
Making a difference to health inequalities
Systematic reviews are uniquely placed to determine if the evidence exists for effective interventions to reduce the global burden health inequity. The acronym PROGRESS (Place, Race, Occupation, Gender, Religion, Education, Socioeconomic status, Social status) provides a relatively straightforward approach to standardizing the way in which determinants of equity are examined.5, 6 Whether or not primary research has addressed inequity or incorporated strategies to improve equity in their programme development, implementation and evaluation, the PROGRESS framework provides an approach for searching for and synthesizing this information through systematic reviews. This provides the ability to highlight essential research gaps and opportunities. While there is a need to improve reporting of studies to include outcomes by PROGRESS categories, some studies may be targeted towards participants in a lower SES area or of a particular ethnic minority. Studies of this nature often also include information about some of the ways that their intervention was tailored towards these groups. This information can be incredibly useful for those attempting to replicate that intervention to address a similar problem and can be extracted from primary studies relatively quickly.
New Cochrane protocols and reviews of interest to health promotion and public health stakeholders from Issues 5–7, 2011 of The Cochrane Library (*denotes CPHG review/protocol)
Reviews
An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes.
Collaboration between local health and local government agencies for health improvement*
Community interventions for preventing smoking in young people (updated).
Consumer-oriented interventions for evidence-based prescribing and medicines use: an overview of systematic reviews.
Hepatitis B vaccination during pregnancy for preventing infant infection.
Interventions for enhancing consumers' online health literacy.
Interventions for improving coverage of child immunization in low- and middle-income countries.
Interventions targeted at women to encourage the uptake of cervical screening.
Nutritional advice for improving outcomes in multiple pregnancies. Pre-employment examinations for preventing occupational injury and disease in workers.
Reduced dietary salt for the prevention of cardiovascular disease.
Strategies for integrating primary health services in low- and middle-income countries at the point of delivery (updated).
Universal school-based prevention programmes for alcohol misuse in young people.
Vitamin D supplementation for prevention of mortality in adults.
Protocols
Environmental and behavioural interventions for reducing physical activity limitation in community dwelling visually impaired older people.
Intermittent iron supplementation for reducing anaemia and its associated impairments in menstruating women.
Interventions for managing absenteeism among health workers.
Interventions for recruiting smokers into cessation programmes.
Knowledge translation strategies for facilitating evidence-informed public health decision-making among managers and policy-makers*
Non-specialist health worker interventions for mental health care in low- and middle-income countries.
Strategies for integrating family planning services with maternal, neonatal and child health and nutrition services.
Workplace pedometer interventions for increasing physical activity.
Synthesis of complex intervention studies
Where appropriate, a meta-analysis is an effective way of pooling data and providing summary estimates of effect. However, for complex interventions, and arguably any intervention, a narrative synthesis is essential either alone (when a meta-analysis is not appropriate or possible) or alongside a meta-analysis, as it allows an exploration of the implementation-related factors and equity considerations mentioned above. To ensure the relevance of the synthesis to policy-makers and practitioners, we suggest that review authors interpret the findings through a practitioner or policy-maker lens, putting themselves in the position of the end users who will inevitably be looking for information to guide a decision, perhaps about allocation of resources or programme implementation.
To this end, some ways of ‘cutting’ the data in terms of organizing the studies may be by setting, intervention type or population group. These are only a few options and it is important to think broadly and creatively at this stage, and the logic model should also provide a useful framework. RAGs or less formal consultation processes can also assist in helping to identify important issues to highlight in the review for informing the decision-making processes of relevant stakeholders.
Conclusion
All intervention research within a public health context is arguably complex by virtue of its design, context or system within which it is operating. Understanding effectiveness cannot be separated from implementation and contextual factors. Standard best practice for reviews of intervention evidence should be to integrate effectiveness at various levels, combined with the implementation and contextual factors within which each programme or intervention is operating. Obviously, this cannot be undertaken unless these data are collected within the primary research or evaluation. However, with relatively small changes or additions to the evidence review process practical, meaningful and rigorous public health evidence can be generated.
We have presented some recommendations for ensuring evidence reviews of such interventions are useful to decision makers. Importantly, the needs of end users should be considered throughout the review process from guiding the scope of the review question through to informing the ways in which the evidence is organized and synthesized. Logic models make processes visually explicit and map the assumed links between interventions and outcomes. Implementation information must be considered simultaneously with effectiveness data and it is possible to extract this during the evidence review process without a large resource investment.
Funding
The CPHG acknowledges the support of the Victorian Health Promotion Foundation (VicHealth) and the Commonwealth Department of Health and Aging, Australia.
