Methods of assessing value for money of UK-based early childhood public health interventions: a systematic literature review

Abstract Introduction Economic evaluation has an important role to play in the demonstration of value for money of early childhood public health interventions; however, concerns have been raised regarding their consistent application and relevance to commissioners. This systematic review of the literature therefore aims to collate the breadth of the existing economic evaluation evidence of these interventions and to identify the approaches adopted in the assessment of value. Source of data Recently published literature in Medline, EMBASE, EconLit, Health Management Information Consortium, Cochrane CENTRAL, Cochrane Database of Systematic Reviews, Health Technology Assessment, NHS EED and Web of Science. Areas of agreement The importance of the early childhood period on future health and well-being as well as the potential to impact health inequalities making for a strong narrative case for expenditure in early childhood public health. Areas of controversy The most appropriate approaches to evaluating value for money of such preventative interventions relevant for UK decision-makers given the evident challenges. Growing points The presented review considered inconsistencies across methodological approaches used to demonstrate value for money. The results showed a mixed picture in terms of demonstrating value for money. Areas timely for developing research Future resource allocations decisions regarding early childhood public health interventions may benefit from consistency in the evaluative frameworks and health outcomes captured, as well as consistency in approaches to incorporating non-health costs and outcomes, incorporating equity concerns and the use of appropriate time horizons.


Introduction
The importance of the early childhood period on future health and well-being is well established. 1,2 Interventions in early life not only have the potential to impact diseases in adult life but also to impact health inequalities experienced throughout the life course. 3 -7 Policy recommendations in the UK and elsewhere have therefore reiterated the importance of increasing public or government expenditure by local and national decision-makers on those in early childhood 6 ,8 as well as more broadly acknowledging the value of investing in preventative interventions. 9 Despite the presence of a strong narrative case for prevention in early childhood, it is important that decisions to fund such interventions are based on systematic and robust assessments of clinical and economic evidence. Resources are limited and decisions to fund an intervention means the opportunity to fund alternative interventions are foregone. Economic evaluation provides a systematic and transparent framework to identify which interventions offer value for money and help inform the choice of competing claims on limited resources. 10 In the UK, methods of health economic evaluation are generally well established when informing health technology assessment decisions. 11 That is, interventions for treating existing conditions. Yet, when it comes to preventative interventions at a population level, there remains less agreement on the most appropriate methods for conducting economic evaluation. 12,13 This is particularly true for public health interventions targeting early childhood given the complex interplay between health, development, education, socioeconomic status and the family environment. 14 -16 The range of methodological challenges of conducting economic evaluations of such early childhood interventions has been highlighted in the literature and includes: appropriate time horizons, measuring and valuing health outcomes, incorporating non-health costs and outcomes, and informing health equity concerns. [17][18][19] These challenges may in-part explain why a number of large-scale early childhood preventative interventions have failed to show cost-effectiveness 13 and why gaps exist between the evidence base and decision-making. 20 Previous literature reviews in a paediatric setting have focussed on economic evaluations of specific intervention categories, such as vaccinations, 21 parenting interventions, 22,23 health promotion, 24 -26 and oral health. 27 Furthermore, these reviews have not limited the results to a specific country or jurisdiction. We consider it important to identify the economic evidence relevant to UK public health decision-makers given the positive and normative reasons why results of economic evaluations may differ across jurisdictions. 28 A literature review of public health intervention decisions made by NICE 29 focussed on those conducted in a UK context yet there was no limitation by age category and the review focussed on NICE guidance and not the wider evidence base.
This systematic review of the literature therefore aims to achieve a number of goals. First, to collate the breadth of the existing economic evaluation evidence of early childhood public health interventions conducted in a UK context. Second, to describe the methods and approaches adopted in the evidence base to highlight consistencies in the demonstration of value for money. Finally, to critically appraise the quality of the evidence base. By doing so, this review seeks to provide researchers and policymakers in the UK details of the relevant economic evaluation evidence, as well as highlighting the methodological challenges and deficiencies in conducting such analyses.

Methods
The protocol was registered with PROSPERO (CRD42021270751) and was conducted and reported in accordance with PRISMA guidelines. 30

Data sources and searches
An initial search strategy was designed in Ovid MEDLINE with the final strategy adapted with relevant subject headings (controlled vocabularies) and search syntax to each of the databases listed below. No language limits were applied, but papers were limited to 2000 onwards to ensure the relevance to the current research and policy deliberations. Details of the full search strategies are contained in Supplementary Material, Supplementary Appendix 1. The

Study selection
Two review authors (P.M. and L.P.) independently conducted title and abstract screening of a random sample of 10% of the retrieved records. A kappa statistic for assessing inter-rater agreement 32 was calculated. Upon the achievement of a kappa statistic of 0.8 or above, one reviewer screened the remaining titles and abstracts. Failure to achieve the required kappa statistic meant a further 10% would be screened by both reviewers until the required score was achieved. Reviewers screened 20% (two screening rounds) before the sufficient kappa statistic was achieved. This process was applied at both the title and abstract screening stage and the full text screening stage. Any discrepancies were resolved by discussion between the two review authors.
Records were included if they reported economic evaluations of public health interventions in the UK. Public health was defined using terms that broadly reflected interventions of health improvement and included terms for wider social determinants of health (Supplementary Material, Supplementary Appendix 1). Evaluations were limited to those of interventions for infants and children with a mean age of 5 years or under at baseline to reflect the infant, toddler and preschool years. Economic evaluations of interventions for ages above 5 years of age were considered for inclusion if they explicitly included a subgroup analysis for those 5 years or under. Those evaluating interventions aimed at pregnant women or for the treatment of existing conditions in infants, children and family members were excluded.
All studies that aimed to inform a value for money assessment of both the cost-and health-related outcomes of an intervention were included. However, the methodological approach taken in the study was stratified into a number of groups conditional on whether they combined the costs and outcomes into a single framework and/or incorporated a comparator intervention.
Cost-effectiveness analyses (CEA) were defined as studies that captured the costs and health outcomes of competing interventions, with health outcomes expressed as either a generic measure of health such as quality-adjusted life years (QALYs), referred to hereafter as 'QALY-based CEAs', or expressed in alternative natural units, referred to hereafter as 'non-QALY-based CEAs'.
Cost-consequence analyses (CCA) were defined as those reporting disaggregated costs and health outcomes of competing interventions; and costbenefit analysis (CBA) that captures the costs and outcomes, both expressed in monetary terms.
Evaluation frameworks in the form of 'performance measures' that consider both costs and outcomes but do not necessarily require a comparative analysis were also included in this review, owing to their inclusion in Public Health England's Health Economic Evidence Resource (HEER) tool. 33 This includes social return on investment (SROI) and return on investment (ROI). Both present a ratio of the monetary returns to the money spent, with the former focussing on the wider costs and benefits to society beyond just healthcare. 13 Sources of grey literature amongst the search results were included in Ovid's HMIC database, which includes literature on health management, health service policies, public health and social care with an emphasis on the UK and the NHS. Grey literature that was found through supplementary searches of previous systematic reviews was deemed eligible for inclusion.

Data extraction and critical appraisal
A de novo data extraction pro forma was used. The extracted information was based on central characteristics of the included studies: intervention, comparator and the population. The type of evaluation framework used was extracted as well as the perspective adopted, and the associated extent of the costs and outcomes (health and non-health) included in the evaluation. The time horizon, use of decision modelling and information on the type of decision model (such as decision tree or Markov model) were also extracted. Finally, the incorporation of any equity considerations in the economic evaluation and the empirical results of the evaluation were extracted. The pro forma can be found in Supplementary Material, Supplementary Appendix 2.
Critical appraisal of the included studies was conducted through the use of the CHEERS checklist, 10 which can be found in Supplementary Material Supplementary Appendix 3.

Review profile
The database search retrieved 16 879 records resulting in 12 592 unique records following deduplication. Of these, 207 full text articles were screened and 58 met the eligibility criteria. Incitation searching of the previous systematic reviews yielded an additional 13 that met the eligibility criteria. In total, 71 articles were included in the synthesis. See Fig. 1 for the PRISMA flow diagram. 34 In the case of three papers, the same evaluation was described in two separate papers: Morrell 35,36 ,Pandor 36 ,37 , and Jacklin 38 and NICE. 39 One paper by Kendrick 41 included two evaluations of differing approaches and interventions. The results of this systematic review are therefore based on 69 individual evaluations.
A little under half of the evaluations were QALYbased CEAs (46%; 32/69). The majority of evaluations were non-QALY-based CEAs (49%; 34/69), with the health outcomes used including life years gained/saved (29%; 10/34), oral health outcomes such as dental caries detected or number of teeth free from decay (15%, 5/34) and cases of a specific disease or condition detected (12%, 4/34). See Table 2 for the list of outcomes. One evaluation (1%) was a CCA 104 and one (1%) a CCA alongside a CEA. 97 Of the studies reporting outcomes in monetary terms, the evaluations identified were SROI (4%; 3/69), CBA (3%; 2/69) with one of the CBAs being conducted alongside a CEA. Finally, four (4%) were not explicit about the type of evaluation used; however, detailed inspecting suggested two of them could be classified as CCA 35 ,42,98 and two as ROI analysis. 76,96 The most commonly reported perspective was the NHS or NHS and personal social services (PSS) (55%, 38/69), which is consistent with the latest NICE methods guidance. 11 A considerable  The time horizons over which the costs and outcomes of the interventions were captured were predominantly one of two categories: those with a short time horizon, i.e. 0-10 years (54%, 37/69), or those with a lifetime horizon that was categorized as 76 years and over (32%, 22/69). The exact number of years over which the costs and outcomes were evaluated was unclear in the case of four evaluations (6%; 4/69) and in the evaluation by Pokhrel, 42 the time horizon differed according to the type of outcome evaluated.
The overwhelming majority of evaluations did not formally incorporate equity considerations (97%; 67/69), see Table 2. Two evaluations (3%) 74,88 considered the cost-effectiveness results across two different social groups. This provided insight into the distribution of health-related outcomes and therefore the cost-effectiveness across social groups. Figure 2 presents the reported results from each study, grouped by the framework used and the intervention group. For QALY-based CEAs, 26 evaluations reported an incremental cost per QALY result. In Fig. 2, the NICE-adopted policy threshold of £20 000 per QALY is represented by the dashed red line in the QALY-based CEA plot. Six additional interventions were not included in Fig. 2: three were considered to be dominant and three were considered to be dominated. See Table 1 for further details. Two evaluations presented the results in Euros per QALYs and were therefore excluded from Fig. 2. The results of the non-QALY-based CEAs and the CCAs are not included as the outcomes of the results differ across evaluations.

Reported value for money of the interventions
Both of the CBAs reported a benefit to cost ratios above 0 (the point at which the intervention is considered value for money). The study by Thomas 70 presented the results by disease category, only those results for bronchopulmonary dysplasia are included in Fig. 2 as this disease had the highest benefit/cost (B/C) ratio. The other disease category B/C ratios reported in Thomas can be seen in Table 1. All of the identified SROIs and ROIs indicated that for every pound spent on the intervention, a return of >£1 would be generated (Fig. 2). For four of the five SROIs and ROIs, the evaluations were conducted without a comparator arm.

Quality assessment
Fourteen papers reported on all aspects of the CHEERS Checklist. In the majority of the categories of the Checklist, reporting was good, including the perspective, time horizon and competing alternatives. However, the value of the reported costs and outcomes was unclear or not reported in 16 evaluations, with 20 of the evaluations with a time horizon over 1 year failing to report the discount rate for costs and outcomes.
Finally, only 29 of the evaluations covered all of the issues of concern in the presentation and discussion of results. The results of the detailed CHEERS checklist are presented in Supplementary Material, Supplementary Appendix 3.

Discussion
The results show the breadth of UK-focussed economic evaluations of early childhood public health interventions reported or discussed in the published literature. The methods adopted in the demonstration of value for money showed a lack of consistency across many aspects including the type of economic evaluation, the health outcomes captured and the perspective adopted. Fourteen papers reported on all aspects of the CHEERS Checklist meaning 55 (80%) were lacking elements required of a well-reported economic evaluation.
Many of the evaluated interventions were deemed value for money from the perspectives taken. Twelve (38%) of the QALY-based CEAs are cost-effective against the NICE policy threshold of £20 000 per QALY. However, the interpretation of some of the results may require particular consideration given the health, economic, political and social context of these studies may have changed between 2000 and present day.
Because of the evaluative framework chosen for many of the other studies a robust statement of value for money of the intervention is not always possible. For example, in non-QALY-based CEAs, an explicit statement of cost-effectiveness is challenging when the outcome is a metric other than a generic measure of health such as the QALY as it is not possible to compare across different health dimensions. Much has been made of the limitations and challenges of using QALYs for paediatric populations 17 ,19 but their use does allow the comparison of interventions across diseases areas as well as the consideration of the displaced resources (or the 'opportunity cost').
All of the interventions evaluated using SROI, ROI or CBA frameworks could be considered value for money as they were deemed to generate more monetary benefits than the costs (having a ratio of >£1 of benefit per £1 of cost in the case of the SROI and ROI). Yet, caution is required when considering these results. None of the SROI or ROI evaluations incorporated the opportunity cost and it was made explicit in only one CBA. 70 The exclusion of such a fundamental aspect of economic evaluations results in an overestimation of the value of the intervention and risks doing more harm than good to the public by neglecting the health foregone through the net effect of spending. Furthermore, four out of five of the SROI and ROI evaluations were conducted without a comparator. The lack of the inclusion of the opportunity cost or a comparator may feed into the previously reported challenges of allocation decision using ROI. 105 The broad range of the types of evaluation and outcomes may reflect the diverse nature and needs of the decision-makers relevant to such interventions. Public health commissioning decisions in the UK are often the responsibility of local commissioners of services, such as local authorities and clinical commissioning groups (CCGs), not national decision-makers such as NICE. Although NICE's public health approach allows for flexibility in the methods, evaluations conducted using the NICE methods guide may fall short of reflecting the challenges faced by CCGs. 106 Although only a minority, a number of evaluations attempted to incorporate the wider social value of the intervention beyond the value to the health care system. A total of 18 evaluations adopted a 'societal perspective' but the results identified a lack of consistency in the included aspects of value. The inclusion of lost productivity to the parent or caregiver (in the form of wages lost) featured heavily in the evaluations, as did incorporating costs falling on special education services and legal services, yet none featured consistently. The implication of such inconsistencies is that value judgements about what 'should' count are falling on the researchers rather than socially legitimate decision-makers. 107 Public health guidance issued by NICE 108 does allow for flexibility in the costs and outcomes considered in an economic evaluation, but the lack of explicit value judgements may facilitate such inconsistencies.
The results showed the most common time horizons were either 0-5 years or those that extended beyond 76 years. Reasons for this appear to be based around whether an intervention was a trial-based evaluation or those that incorporated decision modelling to model the long-term costs and outcomes. Guidance in the economic evaluation literature indicates that time horizons should be long enough to reflect all of the important differences in costs and outcomes between comparators. 10,11 Such horizons may be well defined for patient-focussed health technologies but not for population-focussed interventions that aim to change behaviour, education, housing and so on. Given the evidence linking the social determinants of health and life expectancy, 109 it stands that a lifetime horizon may be more appropriate.
One aspect of relative consistency in the methods was the lack of the formal incorporation of equity considerations. Interventions implemented in early life have considerable potential to disrupt existing inequalities 7 and remain a fundamental reason for targeting these important years. Yet, the formal incorporation of equity does not appear to be common practice in economic evaluation in this setting. There are now a number of approaches to formally incorporate equity considerations into CEAs. 110 The focus of this review was to identify interventions relevant to UK decision-makers. However, there may be important information available in an international context to aid learnings around the use of methods and approaches relevant to the UK. Future research may consider describing the methods and approaches adopted in the global evidence base to highlight consistencies in the demonstration of value for money in those economic evaluations developed for an international context.

Limitations
A limitation is that there may be relevant and uncaptured evaluations in the grey literature. This is evidenced through the identification of evaluations produced by NICE, 38 Social Value UK, 101 Barnardo's 94 and the Joseph Rowntree Foundation, 97 which were not identified in the database search but rather through the literature reviews. The search strategy included the HMIC database and lists grey literature amongst its coverage, hence the decision to include relevant grey literature in the reference searches of systematic reviews. The inclusion of the grey literature identified though the supplementary reference searching does in part explain the high number of studies identified in this way (5 of the 13). We considered it important to include grey literature as it is included in the HEER tool, 33 yet a pragmatic decision was made for the purpose of this review. Future literature reviews may consider searching and identifying a wider range of grey literature sources.
A further limitation was the difficulty posed in defining 'public health' for the purpose of the search strategy. The review focussed on interventions that aimed to improve the health of the infant or child yet health improvement in early childhood may be dependent on lifestyle and environment not merely based on biology and genetics. 6 It stands that a social model of health may have generated different results. A pragmatic decision was made to include health terms and terms to capture the wider determinants of health in the search strategy.

Conclusion
In addition to identifying the breadth of evidence available in the published literature, this review provides an overview of the inconsistent methodological approaches used. The lack of consistency identified in the methods has highlighted a number of issues that may require consideration in the future generation of economic evaluations of similar interventions to aid decision-making. It is hoped the results can provide a foundation to help improve decisionmaking and provide a starting point for methodological developments in the early childhood public health context.

Authors' contribution
PM, SH and GR discussed the direction of the research and the general approach to the systematic review. HF advised on and ran the search strategies. PM and LP screened identified records. PM drafted the manuscript. All authors commented on the early drafts and the final version of the manuscript.

Funding
This paper presents independent research as part of a PhD programme funded by the National Institute for Health Research's (NIHR) Applied Research Collaboration Yorkshire and Humber. The views expressed are those of the author(s) and not necessarily those of the NIHR.