Exploring the role of economics in prioritization in public health: what do stakeholders think?

Abstract

Background: Debates surrounding the use of conventional approaches in public health and the existence of perceived barriers to using the results of economic evaluations have led to questions posed as to how to establish priorities within public health schemes. The aims of this study were therefore to explore the feasibility and validity of economic evaluation techniques in developing priorities within public health programmes and consider the extent to which different presentational approaches are likely to be incorporated into decision-making, from perspectives of relevant stakeholders. Methods: An advisory board, representative of potential users of economic evaluations, was set up to identify preferences for how findings from economic evaluations might be presented to decision makers and to test the impact of different approaches, different outputs and different presentational styles. The board was divided into two groups, each of which was given three hypothetical ‘scenarios’ to consider. The scenarios comprised descriptions of methods and outputs, with costs, effects, target population and context of intervention constant across all scenarios. Results: The perceived validity of estimates of effectiveness was vitally important, along with sufficient information to gauge whether designs were appropriate and to assess implementation practicalities. Cost–benefit analysis and cost–utility analysis were the preferred approaches despite their complexity, although participants required benchmarks to place net-benefit estimates from cost–benefit analyses into context. Conclusion: Further research is required to substantiate and build on these preliminary findings and collaborations between economists and policy makers are needed to develop clear, rigorous and standard guidance relating to economic evaluation, recognizing the diversity of public health strategies.

economic evaluation, prioritization, public health

Introduction

The need to define and measure the value of public health interventions if they are to compete for scarce public resources has long been recognized.^1,2 However, agreement on the approaches to determine the relative worth of public health schemes remains a contentious issue. Wanless, for example, argued that in order to demonstrate the relative merits of public health investments in relation to clinical interventions rigorous economic evaluations of public health schemes should become the norm.³ Wanless advocated that the approach to be employed should be that as advocated by National Institute for Health and Clinical Excellence (NICE) for health technology appraisals, i.e. the use of cost–utility analysis (CUA).³ In recent years, the measurement of health outcomes using quality-adjusted life-years (QALYs) and CUAs have become the accepted approach for the economic evaluation of health interventions.⁴

However, the nature of public health has meant that such an approach might not be the most appropriate. Public health incorporates a range of interventions with a variety of outcomes that cannot adequately be represented by a single metric. They tend to be complex, involving a number of stakeholders and likely to require multi-sector perspectives. As a consequence NICE⁵ has supplemented its reference case for health technology assessment by advocating that for public health: However, the issue remains as to whether these developments are sufficient. Drummond et al.,⁶ for example, identified four challenges to undertaking economic evaluations in public health interventions, which reflect the complexity and coverage of such schemes, as namely measuring outcomes, valuing effects, incorporating equity considerations and identifying inter-sectoral costs and consequences. These are still the subject of much discussion and debate in academic and policy circles and are likely to remain so for some time.

• Additional cost information should be collected to measure costs from a public sector perspective.

• Economic modelling techniques should be used to compensate for the gaps in the existing outcomes evidence in some public health areas.

• Difficulties associated with synthesizing multiple outcomes may necessitate the use of cost–consequences analysis (CCA) instead of CUA.

Alongside these methodological concerns a number of barriers to using the results of economic evaluations in decision-making have been identified.^7–9 Policy makers have struggled to fully utilize such studies due to, for example, variations in methods employed, lack of methodological rigour and reliance on assumptions that do not necessarily reflect actual situations. Furthermore, the outputs produced might not be understandable or are not applicable to other situations, while in some cases, the studies have been undertaken based on narrow research questions, which do not correspond with the agendas of the decision-makers.

As a result of these concerns, and the recognized difficulties of applying the standard economic evaluation techniques to public health programmes, the Welsh Assembly Government (WAG) commissioned this study to explore the use of economic techniques, using the Food and Fitness Implementation Plan¹⁰ as an example. This article discusses some of the emerging findings.

An advisory board of potential users of economic evaluations was set up to identify preferences as to how the findings from economic evaluations might be presented to decision makers and to test the impact of different approaches to economic evaluation, with different outputs, and different presentational styles. The specific aims of the study were to explore the feasibility and validity of using health economic evaluation techniques in developing priorities within public health programmes and consider the extent to which different presentational approaches are likely to be incorporated into decision making, both from the perspective of relevant stakeholders.

Methods

The advisory board was established to ‘test’ and ‘validate’ the use of relevant techniques in establishing priorities within a specific public health programme—the WAG Food and Fitness Implementation Plan. It was considered important to embrace all relevant stakeholder groups, encompassing a variety of perspectives, to secure adequate representation from across the Food and Fitness fields. The delegates were selectively chosen based on personal networks of members of the research team, those identified by WAG and others thought to have an interest and who would want to partake and contribute to the study. The process was deliberately exploratory, with the intention to glean what factors the stakeholders considered to be important in informing the decision-making process, the nature of the information required in relation to effectiveness and ‘cost-effectiveness’ and the extent to which the findings could be regarded as being robust and reliable in contributing to the decision-making process.

The board was divided into two groups, each containing four people, and constructed to be representative of the advisory board (two academics with an expertise in nutrition and exercise, one policy maker from WAG and either a teacher or a school nurse).

Each group was given three hypothetical ‘scenarios’ for their consideration. The scenarios comprised a description of the method and output of an economic evaluation of a public health intervention. The costs, effects, target population and context of the intervention were kept the same across all scenarios. Each scenario described a school-based intervention to increase 11–14-year-old pupils’ physical activity levels. In each case the intervention cost £50 per pupil and increased physical activity levels by 25% over 2 years.

Each scenario described a different methodology for measuring the efficiency of the intervention. Figure 1 summarizes the difference between the scenarios.

Figure 1

Summary of scenarios

The actual scenarios are reported in  Appendix 1.

Each of the groups was given the ‘standard’ scenario (‘intervention’). They were asked to imagine that this was the summary of an economic evaluation of an intervention that they were contemplating investing in. They were given 10 min to consider whether they thought the intervention was a good use of public funds and why, and what extra information may have facilitated their decision-making. This exercise was repeated for the alternative scenarios, with each group being given a different alternative scenario. In the case that the standard and the alternative scenarios were both considered either good or bad use of public funds, the groups were asked to imagine that they had a budget that could be used to implement one (but not both) of the interventions in all schools across Wales. They were given another 10 min to consider which of the interventions they would fund and why.

Each group was then given a third scenario, and again given 10 min to consider whether they thought the intervention was a good use of public funds and why. They were then given another 10 min to consider whether they would prefer to use their budget to invest in this third scenario rather than their preferred of the first two scenarios.

The allocation of scenarios to the groups is summarized in figure 2.

Figure 2

Distribution of scenarios to groups

Results

The first two scenarios assessed by group one were CCA that reported the cost of the intervention and the short-term (2-year) impact. The only difference between the scenarios was that the analysis undertaken in the first scenario focused on changes in physical activity (scenario 1), while the second scenario also reported educational outcomes (scenario 2). In both cases the group were unable to decide whether the intervention represented a good use of public funds. The latter scenario was preferred as it provided more information on the effects of the intervention. However, the group stated a number of pieces of data that they would need before they could decide whether the intervention was a good use of public resources, including:

• An assessment of whether the schools included in the analysis were representative of those in Wales.

• Information on the content of the intervention.

• Estimate of the baseline levels of physical activity in order to determine whether the improvements in physical activity could produce health gains.

• The value of the effects produced, so that they could be directly compared against the cost of the intervention.

The third scenario presented to group one was a cost–benefit analysis (CBA) that reported the cost per participant, the value of long-term health gain and improved earnings, the difference in the value of long-term effects for disadvantaged and non-disadvantaged groups, and the net benefit of the intervention (scenario 5). This scenario was preferred to the previous two, as the monetary valuation of the effect allowed it to be directly compared with the intervention cost. Furthermore, the estimation of lifetime effects was considered a positive, though there was awareness that this type of extrapolation was difficult and subject to uncertainty.

However, despite these improvements in the evidence provided by scenario 5, the group were still unsure whether the intervention represented a good use of public resources. As with scenarios 1 and 2, there were still calls for more information on the content of the intervention and baseline levels of physical activity. There were concerns whether the effects observed were attributable to the intervention, with participants looking to the intervention description rather than the research design to support the validity of the findings. In addition questions were posed as to whether the study was well designed, despite having positive evaluation findings, or whether it was practical to implement locally, which could not be established from the research findings and design alone. Furthermore, participants were unsure whether the net benefit of c£1200 represented good value for money. They wanted to know of the net benefit produced by alternative interventions before deciding whether this level of net benefit was sufficient.

Group two were given the same first scenario as those in group one (scenario 1: a CCA that reported the cost of the intervention and the short-term (2-year) impact on physical activity levels). They were similarly unsure whether the intervention represented a good use of public resources. There were concerns that the effects identified were not attributable to the intervention and that other factors may be explaining the observed improvements in physical activity. However, this may have been the result of a lack of understanding of the research design employed, as despite the scenario stating that a randomized controlled trial was employed; some participants were concerned about the lack of a control group.

The second scenario presented to group two was a CBA that reported the cost per participant, the value of long-term health gain, and the net benefit of the intervention (scenario 4). While this scenario was preferred to the first scenario they assessed, the group were still unsure whether the intervention represented a good use of public resources, despite the intervention clearly producing a net benefit. There was still concern whether the effects observed could be attributed to the intervention. There were also concerns that the average net benefit might not be observed for all participants, and more data were requested on the range of effects and the characteristics of those that did not benefit from the intervention.

The third scenario considered by group two was a CUA that reported the cost per participant, the average number of QALYs gained, the average cost per QALY gained, and the NICE cost per QALY gained threshold (scenario 3). There was agreement that this scenario was the most convincing of the three and that the intervention represented a good use of public resources. The ability to compare the estimate of the cost per QALY gained against the NICE threshold provided reassurance that the intervention was value for money. However, there was some concern that the very small number of QALYs gained (0.01) may be interpreted as only a small effect.

Discussion

There is little evidence on the impact that different types of economic evaluation have on the likelihood that policy makers will invest in an intervention. This study engaged the users of such research, including policy makers, researchers and practitioners, to determine which elements of an economic evaluation were most likely to impact on the likelihood that an efficient intervention was adopted and how the findings should be presented. It was encouraging that although there were concerns about some of the economic approaches presented, in general, economic information was welcomed and wanted.

The following key findings emerged:

• The perceived validity of the estimates of effectiveness on which the economic analysis built is important. In instances when participants were unconvinced about the quality of the intervention, the quality of the research design was insufficient to convince participants of the validity of the effect estimate. Information about the design of the intervention is required so that decision makers can ascertain that the research design was appropriate for the intervention, also to allow for consideration of the practicalities of implementing the intervention locally.

• More information was generally considered better. Scenarios were preferred when the perspective of timeframe of the analysis was broadened. However, broadening the scope of the analysis alone was insufficient to convince participants of the efficiency of an intervention.

• Cost–effectiveness analysis (CEA) or CCA were considered an insufficient basis on which to assess the efficiency of an intervention, as the estimates of cost and effect were not directly comparable. Despite their added complexity and lower transparency, CBA and CUA were preferred.

• An external definition of ‘good’ is required for decision-making. Despite being presented a CBA that concluded that an intervention produced a net benefit, participants wanted estimates of the net benefit of other interventions against which to benchmark this result and determine whether the net benefit was ‘good’. The only instance in which participants concluded an intervention was a good use of public funds was when they could compare the results of the analysis (cost per QALY gained) against an external benchmark (the NICE cost per QALY gained threshold).

This study has a number of implications for the economic evaluation of public health interventions. First, economic evaluation needs to build on good quality, convincing analysis of effectiveness. Second, CBA and CUA are preferred to CEA and CCA. Third, policy makers need assistance interpreting the outputs of economic analysis in order to understand whether an intervention is efficient and useful for decision-making purposes.

Furthermore, methods employed in other branches of economics might provide insight into how the challenge of multiple outcomes from public health interventions can be dealt with. The principal frameworks developed for the appraisal of environmental and transport interventions, for instance, revolve around the notions of CBA and multiple criteria decision analysis (MCDA).¹¹ While CBA has been utilized within public health, MCDA has only had a limited role. For instance, the USA by the Preventive Services Task Force ranked a list of clinical preventative interventions based on their cost–effectiveness and clinical preventable burden.^12,13 MCDA’s ability to evaluate interventions across multiple outcomes, while formally incorporating decision makers into the evaluation process, makes it an attractive approach in overcoming some of the challenges facing the economic evaluation of public health interventions.

The subjective wellbeing approach (SWB) is also attracting attention as a potential series of measures to support policy analysis and could provide a more holistic approach to capturing and valuing the outcomes associated with public health interventions. It is important, however, to note that the approach is still in its infancy and a number of methodological questions need to be answered before this approach can be used to generate evidence to inform policy making.

A key conclusion pertinent to whether CBA is useful in evaluating public health interventions is that study participants were not sure whether a positive net benefit reflected a good enough outcome! It was suggested that more information was required about the net benefit of alternative programmes and examples of other CB ratios as benchmarks. There was a suggestion that this may be due to the fact that there was a feeling that an alternative programme may offer a larger net benefit, and therefore, there was a desire to have other CB ratios to compare against, although whether this would be a sufficient criterion to establish priorities remains open to question. If policy makers were furnished with the findings of CBAs for different possible programmes all with positive net benefits, it is likely that there may be issues about comparability of the studies that would have to be considered.

However, it is important to understand the limitations of the analysis undertaken. At best the results can be considered indicative. The small number of participants, as well as the limited number of scenarios tested, means that the results should be treated with caution. The study was conducted within a very tight timescale, with limited time for discussion and explanation in each of the advisory group meetings; more time for explanation and discussion might have produced different results. Furthermore, there was difficulty in maintaining the attendance of Advisory Board members, with some simply not attending, whilst others sent a replacement, having implications for continuity of the ideas discussed in previous meetings. Engagement is a key issue for the development of this agenda. If real progress is to be made with the application of health economics to public health, a relationship needs to be developed between research and policy that is maintained over a long period. In many ways, this study was the start of a process, but much more work needs to be done.

There is a spectrum of public health interventions ranging from the purely private to the totally public good: all public health programmes however have some multidimensional, inter-temporal and social aims that distinguish their economic effects from say, treatment in acute care. Further research, in conjunction with policy makers, is required to substantiate and build on the findings obtained in this study. For instance, one possibly fruitful line of enquiry would be to employ a discrete choice experiment approach to determine policy makers’ preferences for interventions assessed using different methods. It would be interesting for example, to see to what extent the preferences about the way economic evaluation information is presented to decision makers differs in the public health arena compared to other areas of health care—to determine whether there are broader lessons to be learned about how economic evaluation findings are understood and therefore utilized in decision making.

The rigour and attention devoted to the assessment of health technology assessment needs to be applied with similar fervour to public health interventions and programmes and economic evaluation should be integral to every public health intervention and not bolted on as an after-thought. Economists and policy makers alike must strive to further develop clear and standard guidance, recognizing the diversity of public health strategies. Furthermore, public health initiatives should be regarded as an investment in the health of the nation and not be expected to generate relatively short payback periods, and fit into narrow definitions and perspectives as to what constitutes cost–effectiveness.

Funding

This article has resulted from a project funded by the Welsh Assembly Government, Cardiff, Wales, UK.

Conflicts of interest: None declared.

Key points

• Economic evaluation needs to build on good quality, convincing analysis of effectiveness of public health programmes.

• Second, CBA and CUA are preferred to CEA and CCA.

• Policy makers need assistance interpreting the outputs of economic analysis in order to understand whether a programme is relatively efficient and useful for decision-making purposes.

• Further research is needed whereby economists and policy makers collaborate to develop clear, rigorous and standard guidance relating to economic evaluation, recognizing the diversity of public health strategies.

Appendix 1

Contents	Three 1-h, teacher-delivered classroom sessions over 3 weeks, focussing on decreasing television viewing, decreasing consumption of high-fat foods, increasing fruit and vegetable intake, and increasing moderate and vigorous physical activity
Setting	Twelve mixed schools in deprived areas of Wales
Participants	11–14-year-old pupils
	Scenario 1	Scenario 2	Scenario 3	Scenario 4	Scenario 5
Methods	Schools were randomly allocated to receive the intervention or to continue with normal syllabus. Interviews were undertaken with pupils to measure levels of physical activity before and 2 years after the intervention in each of the schools. Interviews were undertaken with school staff to measure the time and materials required to prepare for and deliver the intervention.	Schools were randomly allocated to receive the intervention or to continue with the normal syllabus. Interviews were undertaken with pupils to measure levels of physical activity and educational attainment before and 2 years after the intervention in each of the schools. Interviews were undertaken with school staff to measure the time and materials required to prepare for and deliver the intervention.	Schools were randomly allocated to receive the intervention or to continue with the normal syllabus. Interviews were undertaken with pupils to measure levels of physical activity and educational attainment before and 2 years after the intervention in each of the schools. Using studies of the long-term health gains associated with physical activity, the likelihood of avoiding a coronary heart disease (CHD) event over the lifetime of participants as a result of the observed changes in physical activity was predicted. Data on the impact of avoided CHD events on quality and length of life were used to predict the number of QALYs gained as a result of the intervention. Interviews were undertaken with school staff to measure the time and materials required to prepare for and deliver the intervention.	Schools were randomly allocated to receive the intervention or to continue with the normal syllabus. Interviews were undertaken with pupils to measure levels of physical activity and educational attainment before and 2 years after the intervention in each of the schools. Using studies of the long-term health gains associated with physical activity, the likelihood of avoiding a CHD event over the lifetime of participants as a result of the observed changes in physical activity was predicted. The QALYs gained by pupils from disadvantaged areas were valued using the upper bound of the values that the NICE place on a QALY (£30 000). The QALY gained by pupils from non-disadvantaged areas were valued at the lower bound of the values that NICE place on a QALY (£20 000). Interviews were undertaken with school staff to measure the time and materials required to prepare for and deliver the intervention.	Schools were randomly allocated to receive the intervention or to continue with the normal syllabus. Interviews were undertaken with pupils to measure levels of physical activity and educational attainment before and 2 years after the intervention in each of the schools. Using studies of the long-term health gains associated with physical activity, the likelihood of avoiding a CHD event over the lifetime of participants as a result of the observed changes in physical activity was predicted. The QALYs gained were valued using the upper bound of the values that the NICE place on a QALY (£30 000). The public sector costs saved due to avoided CHD events were valued at the average rate of a treated CHD event (£2000). Data on the relationship between educational attainment and lifetime earnings were used to value improvements in educational attainment. Interviews were undertaken with school staff to measure the time and materials required to prepare for and deliver the intervention
Results	The intervention cost £50 per pupil to deliver. On average, 2 years following intervention, physical activity levels of pupils that received the intervention were 25% higher than pupils that did not receive the intervention.	The intervention cost £50 per pupil to deliver. On average, 2 years following the intervention, the physical activity levels of those pupils that received the intervention were 25% higher than those pupils that did not receive the intervention. Among the pupils that received the intervention, the increase in physical activity levels among those pupils who lived in disadvantaged areas was 10% higher than other pupils. On average, 2 years following the intervention, the educational attainment levels of those pupils that received the intervention were 10% higher than those pupils that did not receive the intervention.	The intervention cost £50 per pupil to deliver. On average, those participating in the intervention were predicted to gain 0.01 QALYs more over their lifetimes than those who did not receive the intervention. On average, the cost per QALY gained as a result of the intervention was £5000, significantly below the NICE’s £20 000–30 000 per QALY. NICE suggest that interventions with a cost per QALY gained below this threshold are good value for money.	The intervention cost £50 per pupil to deliver. On average, the lifetime benefit to those who participated in the intervention due to avoided CHD events was £230. On average, the net benefit of the intervention was £180 per participant.	The intervention cost £50 per pupil to deliver. On average, the lifetime benefit to those who participated in the intervention due to avoided CHD events was £230. This benefit was the same for those who lived in disadvantaged areas and those who lived other areas. On average, the public sector costs saved due to avoided CHD events was £50. This benefit was the same for those who lived in disadvantaged areas and those who lived other areas. On average, the lifetime benefit to those who participated in the intervention due to improved earnings was £1000. For those who lived in disadvantaged areas, the benefits due to improved lifetime earnings was £1500. On average, the net benefit of the intervention was £1230 per participant. For those who lived in disadvantaged areas, the net benefit of the intervention was £1730.