Examining the use of economic evaluations in health-related humanitarian programmes in low- and middle-income countries: a systematic review

Abstract

The costly nature of health sector responses to humanitarian crises and resource constraints means that there is a need to identify methods for priority setting and long-term planning. One method is economic evaluation. The aim of this systematic review is to examine the use of economic evaluations in health-related humanitarian programmes in low- and middle-income countries. This review used peer-reviewed literature published between January 1980 and June 2018 extracted from four main electronic bibliographic databases. The eligibility criteria were full economic evaluations (which compare the costs and outcomes of at least two interventions and provide information on efficiency) of health-related services in humanitarian crises in low- and middle-countries. The quality of eligible studies is appraised using the modified 36-question Drummond checklist. From a total of 8127 total studies, 11 full economic evaluations were identified. All economic evaluations were cost-effectiveness analyses. Three of the 11 studies used a provider perspective, 2 studies used a healthcare system perspective, 3 studies used a societal perspective and 3 studies did not specify the perspective used. The lower quality studies failed to provide 7information on the unit of costs and did not justify the time horizon of costs and discount rates, or conduct a sensitivity analysis. There was limited geographic range of the studies, with 9 of the 11 studies conducted in Africa. Recommendations include greater use of economic evaluation methods and data to enhance the microeconomic understanding of health interventions in humanitarian settings to support greater efficiency and transparency and to strengthen capacity by recruiting economists and providing training in economic methods to humanitarian agencies.

Economic evaluation, humanitarian crisis, cost-effectiveness, disability-adjusted life years, public health interventions

Key Messages

• Economic evaluations are crucial in guiding humanitarian operational and policy decision-making, improving aid efficiency and strengthening accountability.

• This review identified only 11 studies using economic evaluations of health interventions in humanitarian crises in low- and middle-income countries. Few of these studies applied a societal perspective, most studies used aggregate programmatic costs data and the quality of the studies was mixed.

• Studies with lower quality scores focused on the effectiveness data and included limited costing data.

• There should be a greater focus on the use of economic evaluations of health interventions in humanitarian crises and capacity strengthening on the collection and application of economic evaluation data among agencies and researchers working in humanitarian crises.

Introduction

Humanitarian crises include complex emergencies such as armed conflict, famines, major epidemic outbreaks and natural disasters such as earthquakes. The health challenges faced by crisis-affected populations, including refugees and internally displaced persons, include high mortality and morbidity rates due to communicable diseases, malnutrition, non-communicable diseases (NCDs), poor mental health and poor reproductive and sexual health. It was estimated that 201 million individuals needed humanitarian protection and aid in 2017, with the vast majority living in low- and middle-income countries (LMICs) (Global Humanitarian Assistance, 2018). In the same year, a record of US$27.3 billion was spent on humanitarian assistance (compared with US$18.4 billion in 2013), with much of it focused on conflicts in the Middle-East—most notably Syria and Yemen (Global Humanitarian Assistance, 2018). With increases in the number of people requiring humanitarian aid, rising financial costs for providing humanitarian health services (particularly NCDs), and resource constraints in aid availability, there is a need to maximize the efficiency of the use of humanitarian aid and ensure accountability and transparency and better planning and priority setting in its use (Spiegel et al., 2014; Spiegel, 2017).

The scarcity of resources in humanitarian settings becomes an economic problem of how to allocate limited resources. One method to help address this problem is through economic evaluations. An economic evaluation is ‘the comparative analysis of alternative courses of action in terms of both their costs and consequences’ (Drummond et al., 2015). The main types of full economic evaluation include cost-effectiveness (sometimes called cost-utility) and cost-benefit analyses. The differences between approaches lie in details and sometimes are more cosmetic than substantial. Broadly, cost-effectiveness/cost-utility analysis involves comparison of costs and outcomes, with outcomes expressed as natural units (e.g. cases averted) or, more commonly, in generic measures of health outcome reflecting both mortality and morbidity [e.g. quality-adjusted life years (QALYs) or disability-adjusted life years (DALYs)]. In cost-benefit analyses, outcomes are typically expressed in monetary units. Whatever the particular method used, analyses can help to inform whether more effective but more costly alternatives are worth the commitment of additional resources. This requires assessing the benefits gained against what could otherwise be gained with other uses of those resources (i.e. to opportunity costs) (Drummond et al., 2015). A full description of the types of economic evaluations is listed in Supplementary Data 1. Economic evaluations can therefore support policy-makers and donors to better understand which health interventions offer best value from the limited resources available, which can inform immediate spending decisions, and support long-term budget planning decisions, as well as efforts to generate additional resources for humanitarian crises. In addition, there is a recognition of the value of integrating economic evaluations into the current humanitarian landscape where there are accountability and transparency deficits (Blanchet et al., 2017; Spiegel, 2017; Roberts, 2018).

A number of systematic reviews have been conducted on evidence on the effectiveness of health interventions in humanitarian crises (Hall et al., 2011; Kimbrough et al., 2012; Tol et al., 2014; Ramesh et al., 2015; Ruby et al., 2015; Smith et al., 2015; Blanchet et al., 2017; Singh et al., 2018). However, to the best of our knowledge, there has been no systematic examination of the use of economic evaluations of health-related interventions in humanitarian settings. This systematic review aims to examine the use of economic evaluations in health-related humanitarian programmes in LMICs. The objectives were to: (1) analyse the use of economic evaluation methods (cost-effectiveness, cost-utility and cost-benefit analyses) of health interventions in humanitarian programmes and (2) assess the quality of evidence using economic evaluation methods.

Materials and methods

This systematic review follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Moher et al., 2009) and the Centre for Reviews and Dissemination (CRD) guidance on systematic reviews of economic evaluations (Centre for Reviews and Dissemination, University of York, 2009).

Search strategy

This review focused only on peer-reviewed literature published between 1 January 1980 and 30 June 2018 (as studies prior to 1980 were considered to be very few in number and of lower quality) (Blanchet et al., 2017). This review used the following four main electronic bibliographic databases, which is considered of great relevance to the subject area: Medline, Embase, GlobalHealth and EconLit. The search strategy consisted of search terms related to: (1) humanitarian crises, (2) LMICs (as the vast majority of crisis-affected populations lives in LMICs), (3) public health interventions and (4) economic evaluations. The search terms for the first three topics were obtained from the Humanitarian Health Evidence Review (Blanchet et al., 2015). For the economic evaluation search terms, the UK National Health Service (NHS) Economic Evaluation Database (EED) search strategy was utilized with some modifications to incorporate characteristics from studies conducted in LMICs. The NHS EED is a database containing abstracts of full economic evaluations of health technologies (Centre for Reviews and Dissemination, University of York, 2009). The NHS EED search strategy contained search terms related to QALY, whereas DALYs are more often used in LMIC settings; thus, search terms related to DALYs were incorporated into the search strategy (Glanville et al., 2009). The NHS EED strategy was chosen for its high sensitivity and specificity (Glanville et al., 2009). A detailed list of all search terms is presented in Supplementary Data 2.

Inclusion and exclusion criteria

Table 1 lists the criteria used to select and exclude studies for this review.

Table 1

Inclusion and exclusion criteria

Category	Included	Excluded
Population of interest	Populations that are affected by humanitarian crises, armed conflict, natural and man-made disasters and early recovery in low- and middle-income countries. The low- and middle-income countries were determined using the 2018 World Bank classification	Studies that focus on humanitarian crises in high-income countries Studies centred on asylum seekers or refugees in high-income countries Studies related to military combatants and military veterans
Intervention	Any health-related intervention (i.e. communicable disease, NCDs, nutrition, sexual and reproductive health, mental health and psychosocial support, injury and rehabilitation, water, sanitation and hygiene)	Studies that do not specify health interventions
Types of studies	Full economic evaluations such as cost-effectiveness, cost-utility, cost-benefit and cost minimization. The study design could be a randomized controlled trial or modelling studies	The following studies were excluded: partial economic evaluations, review papers, studies examining preparedness and resiliency not linked to an intervention and studies not linked to any health outcomes in a humanitarian context
Outcomes and outputs of interest	Primary outcomes (e.g. incremental cost-effectiveness ratio) Secondary outcomes (e.g. cost per case averted) Primary outputs (e.g. cost)	Costing-only studies
Data type	Studies must try to relate cost to outcome data	Costing-only studies
Date of publication	1 January 1980 to 30 June 2018
Publication language	English	Other languages

A humanitarian crisis is defined as ‘a serious disruption of the functioning of a community or a society causing widespread human, material, economic or environmental losses which exceed the ability of the affected community or society to cope using its own resources, necessitating a request to national or international level for external assistance. The disaster situation may either be man-made or a natural occurrence’ (Blanchet et al., 2015). Other key definitions for public health interventions, and economic terms, are listed in Supplementary Data 1. Only studies conducted in LMICs, as determined by the World Bank 2018 classification, were included because the vast majority of civilians affected by humanitarian crises lives in LMICs, and availability and use of resources differ significantly from those in high-income countries (Blanchet et al., 2015; World Bank Group, 2018). Studies related to post-conflict settings or with armed combatants/military veterans were not included as these studies are not representatives of the unique environment or civilian populations in crisis settings. In addition, for eligibility, studies needed to have conducted an economic evaluation in a health-related humanitarian programme.

Partial economic evaluations, such as costing studies, were deemed to be ineligible for inclusion in the study. This decision was made to fully understand and discuss the quality and quantity of full economic evaluations that are conducted in health-related humanitarian programmes. Full economic evaluations provide a value for money analysis by comparing the cost and outcomes of two or more interventions, thus allowing an answer to questions related to efficiency, whereas partial economic evaluations do not assess efficiency (Napper and Newland, 2014).

Data screening and extraction

The data screening and extraction was performed in a five-stage process by the authors LAM and NSS. The first stage consisted of downloading and importing search results from the four databases into a reference management software. In the second stage, duplicates were removed. The third stage entailed screening the titles and publication type for eligibility. Records were excluded if they were not peer-reviewed journal articles or did not specify an economic study on a health intervention in a humanitarian setting. The fourth stage involved screening abstracts for eligibility. The fifth stage included a full-text assessment for eligibility. The final stage assessed whether the study was a full or partial economic evaluation. A full economic evaluation compared both the costs and outcomes of two or more interventions requiring a measurement and valuation of both costs and outcomes (Napper and Newland, 2014). A partial economic evaluation considered costs or outcomes but did not involve a comparison between interventions nor did it relate costs to benefits (Napper and Newland, 2014). The full economic evaluations were determined to be eligible for data extraction and quality assessment.

The data were extracted from the final selected studies into a database in Microsoft Excel. The data extraction fields were broadly classified into the following three main categories: study characteristics, economic evaluation methodology and findings (Centre for Reviews and Dissemination, University of York, 2009). The study characteristics included: study authors, year, study country, title, study objective/question, setting, population type, humanitarian crisis type and public health intervention type. The economic evaluation methodology included: economic evaluation type, perspective, time horizon, discounting, comparator, analytical approach, effectiveness data, cost data, monetary benefit and utility valuation. The findings were: key results, authors’ conclusions and quality assessment.

Quality assessment

The following three quality assessment tools were reviewed to determine the best tool for assessing full economic evaluations: the CHEC-list (Evers et al., 2005), the ‘Drummond’ 35-point checklist (Drummond and Jefferson, 1996) and the modified ‘Drummond’ 36-point checklist (Centre for Reviews and Dissemination, University of York, 2009). The CHEC-list was deemed to be unsuitable for this review as it primarily focuses on clinical trials and does not include questions related to productivity changes, an important factor for humanitarian settings. Although the Drummond 35-point checklist was widely used, the modified 36-point Drummond checklist was chosen as the quality assessment tool because it contained an additional item assessing the generalizability of results, making it the most comprehensive checklist for this review. The remaining 35 questions were the same questions found on the original Drummond checklist.

A quality scoring system was developed based on the 36-item checklist and grouped into the following three broad categories: study design, data collection and analysis and interpretation of results. Each question under each category was given equal weightage. The study design contained 7 questions, the data collection included 14 questions and the analysis and interpretation of results had 15 questions. A full list of the quality assessment questions are presented in Supplementary Data 3.

Data analysis

Descriptive thematic analysis was used, guided by the study objectives and extraction sheet criteria (e.g. study characteristics, evaluation methods, analytical perspective and approach). This was conducted independently by two authors, and any discrepancies were resolved through discussions with the senior author. A meta-analysis was deemed inappropriate as the studies reporting a range of interventions and outcomes were not consistent for a statistical procedure. In addition, a meta-analysis did not meet the study objectives. Instead, a narrative synthesis approach was used to analyse the data (Petticrew et al., 2013).

Results

Identification and selection of studies

A total of 10 746 studies were extracted from four peer-reviewed databases (Medline 1997; Embase 3944; GlobalHealth 1594; EconLit 3211). Figure 1 displays the PRISMA flow diagram.

Figure 1.

PRISMA flow diagram

After removing duplicates, a total of 8127 studies were screened by their titles. After the screening of title and publication type, a total of 210 articles were screened for their abstracts. This stage resulted in 154 articles being excluded because studies were conducted in a high-income country, not in a humanitarian setting, were published in a different language or did not include a health intervention. The remaining 56 studies went through a full-text assessment where 30 articles were excluded for the following reasons: the study was not written in English, focused on refugees in a high-income setting, did not have an intervention or did not meet criteria for a humanitarian setting. The remaining 26 studies included 11 economic evaluations that met inclusion criteria and 15 costing studies that were excluded. Although partial economic evaluations such as costing studies did not meet inclusion criteria, the costing-only studies are listed in Supplementary Data 4 as they might be of interest to researchers.

Study characteristics

Over a span of almost 20 years, 11 economic evaluations meeting eligibility criteria were published and the study characteristics are summarized in Table 2.

Table 2

Study characteristics of eligible economic evaluation studies (N = 11)

Study authors (year)	Study country	Population type	Humanitarian crisis type	Type of public health intervention	Type of economic evaluation	Perspective	Analytical approach	Measure of outcome
Naficy et al. (1998)	Malawi	Refugee	Conflict	Cholera treatment and vaccination	Cost-effectiveness	Provider	Decision tree	Cost per cholera case prevented Cost per cholera death averted
Griekspoor et al. (1999)	Sudan	General population	Conflict	Treatment of visceral leishmaniasis	Cost-effectiveness	Not specified	Not specified	Cost per DALY averted
Gosselin et al. (2011)	Haiti	General population	Natural disaster	Orthopaedic trauma	Cost-effectiveness	Provider	Not specified	Cost per DALY averted
Tekeste et al. (2012)	Ethiopia	General population	Natural disaster	Treatment for SAM	Cost-effectiveness	Societal	Not specified	Cost per cured case
Deboutte et al. (2013)	Democratic Republic of Congo	Pregnant women	Conflict	EmOC	Cost-effectiveness	Not specified	Not specified	Cost per HALE gained
Sardar et al. (2013)	Zimbabwe	General population	Outbreak	Cholera vaccination, treatment, sanitation and health promotion	Cost-effectiveness	Not specified	Mathematical modelling	Cost per case averted
Gargano et al. (2015)	Somalia	Internally displaced population	Conflict	Rotavirus immunization for diarrheal disease	Cost-effectiveness	Healthcare system	Decision tree	Cost per DALY averted
Gargano et al. (2015)	Somalia	Refugee	Conflict	Vaccination for ARI (pneumonia)	Cost-effectiveness	Healthcare system	Decision tree	Cost per DALYs averted
McBain et al. (2016)	Sierra Leone	General population	Conflict	Mental health—Youth Readiness Intervention	Cost-effectiveness	Societal	Markov modelling	Cost per QALY gained
Gargano et al. (2017)	South Sudan	Refugee	Conflict	Vaccination for ARI (pneumonia)	Cost-effectiveness	Provider	Decision tree	Cost per DALYs averted
Howard et al. (2017)	Pakistan	Refugee	Conflict	Targeted IRS	Cost-effectiveness	Societal	Not specified	Cost per DALYs averted

SAM, severe acute malnutrition; EmOC, Emergency Obstetric Care; HALE, health-adjusted life expectancy; ARI, acute respiratory infection; IRS, indoor residual spraying.

The oldest study was published in 1998, whereas the latest study was published in 2017. Of the 11 studies, 9 studies were published in the last decade. Of the 11 studies, 9 studies were conducted in African countries, 1 study was conducted in Haiti and the remaining 1 study was conducted in Pakistan. Most studies focused on conflict-affected settings, two studies focused on a natural disaster and one study focused on a disease outbreak setting. Seven studies focused on communicable diseases: one study on rehydration therapy for cholera, one study on an intervention for visceral leishmaniasis, one study on comparing four cholera control programmes, one study on adding the rotavirus vaccine, one study on Haemophilus influenzae type B conjugate (Hib) vaccine and pneumococcal conjugate vaccine (PCV) for acute respiratory infections, one study on indoor residual sprayings for malaria and one study on the Hib vaccine and PCV vaccine for acute respiratory infections. There were one study on community-based therapeutic care for nutrition, one study on short orthopaedic missions for injury and rehabilitation services, one study on a mental health behavioural intervention and one study on obstetric care. Of the 11 studies, 8 studies noted that the compared health intervention was cost-effective, one study noted that the intervention was not cost-effective, and the conclusions from the remaining two studies were unclear. The key findings from the 11 economic evaluation studies are summarized in Supplementary Data 5.

All of the identified economic evaluations used cost-effectiveness analyses, and they used various perspectives of cost-effectiveness analysis (Table 2). Of the 11 studies, 3 studies used a healthcare provider prospective, 2 studies adopted a healthcare system perspective, 3 studies used a societal perspective and 3 studies did not specify the perspective used, which is considered poor practice in economic evaluation. The costs of health care are grouped into direct medical (e.g. medicines and health service provider costs), direct non-medical (e.g. transport costs to patient and health facility), indirect medical (e.g. costs of health care not directly related to interventions under evaluation, such as health workers in a health facility but not directly involved in the intervention) and indirect non-medical costs (e.g. management and supervisory costs), patient costs (e.g. out-of-pocket costs) and productivity costs (e.g. loss of income due to disability and premature death). The issue of which costs to include in an economic evaluation depends on the perspective of the analysis. Three studies used a provider perspective and accounted for the direct and indirect medical costs incurred by the humanitarian agency providing the services. The health system perspective includes direct and indirect costs related to medical services and direct non-medical costs within the scope of the health system (e.g. medicine transport costs and subsequent use of health care at other facilities), and two studies that used a healthcare system perspective compared the cost-effectiveness of a supplemental humanitarian programme with that of a national health programme. The societal perspective is the most holistic approach and includes all direct and indirect medical and non-medical provider and health system costs irrespective of who incurs the costs (e.g. patient transport costs to facilities and productivity loss such as lost earning by the patient), and three studies using a societal perspective reported costs to the healthcare system and also productivity costs to the society as a result of the health condition.

The analytical approach also varied among the 11 studies (Table 2). Four studies used decision tree analysis, one study used a Markov model, one study used a mathematical model and five studies did not specify an analytical approach. There were also differences in the way the outcome measures were reported. Seven studies reported outcomes as cost per DALY averted or health-adjusted life expectancy or QALY gained, whereas the remaining studies reported outcomes as cost per case averted, or cost per death averted.

Quality assessment findings

The quality of studies assessed on the study design, data collection and analysis and interpretation of results shows that most studies reported comprehensively on the study design techniques but lacked quality in data collection and interpretation of the results (Table 3).

Table 3

Quality assessment scores

Studies	Study design (n = 7)	Data collection (n = 14)	Analysis and interpretation of results (n = 15)
Naficy et al. (1998)	6	6	12
Griekspoor et al. (1999)	5	8	11
Gosselin et al. (2011)	2	4	2
Tekeste et al. (2012)	5	5	4
Deboutte et al. (2013)	5	6	9
Sardar et al. (2013)	2	5	3
Gargano et al. (2015)	6	11	15
Gargano et al. (2015)	6	14	15
McBain et al. (2016)	6	14	12
Gargano et al. (2017)	4	10	10
Howard et al. (2017)	7	12	15

n = the total number of questions under each section.

For study design, 8 of the 11 studies were sufficiently well designed, scoring 5/7 for the study design criteria. Only one study scored a full 7/7, and three studies scored 2/7 or 4/7 on the quality scoring metrics. The low-quality studies lacked the justification required for choosing a cost-effectiveness analysis or did not state the importance and rationale for the interventions chosen.

The scoring on the quality of data collection recorded 3 of the 11 studies as high quality. These three studies scored 12/14 or higher on the data collection methods and were able to justify the sources of effectiveness and cost data as well as clearly state the assumptions and methodology required to document the economic costs. Six studies that exhibited poor quality scored between 4/14 and 8/14, tended to exclude information on the source of effectiveness, the valuation of health states, price adjustment and currency conversion and did not report resources separately from unit costs. These six studies lacked details about how data were synthesized and the reason behind the analytical model used. Two studies were of moderate quality, scoring 10/14 or 11/14 on the data collection techniques.

The quality assessment on the analysis and interpretation of results evaluated the studies on the choice of parameters such as discount rates, as well as the sensitivity analysis and results communicated in the studies. Of the 11 studies, 3 studies exhibited high quality by meeting all 15/15 criteria, 4 studies exhibited medium quality and 4 studies exhibited low quality. The studies with low quality failed at noting and justifying a time horizon of costs and discount rates and exhibited a lack of reporting on sensitivity analyses. Only 7 of the 11 studies addressed the generalizability of results in their discussion. Studies that scored good quality in the data collection section also performed well in the analysis and interpretation.

Discussion

Study findings

To the best of our knowledge, this review is the first of its kind to systematically examine the use of economic evaluation studies of health interventions in humanitarian settings. The key finding of this review is that there is limited evidence of full economic evaluations of health interventions in humanitarian programmes. This suggests that opportunities are being missed to help guide operational and policy decision-making and to improve the efficiency and accountability of humanitarian health programming.

Overall, 11 studies represent good practices in identifying and using outcomes data, but they contain some limitations in the application of economic data. The study design assessment suggested that most studies justified the choice of economic evaluation and interventions for the study design. The variance in quality between studies primarily occurred in the data collection and analysis techniques. The quality of studies is important as these studies provide effectiveness and cost data that can be utilized for resource allocation decisions and thus for budgetary considerations for humanitarian programmes.

The perspective for the evaluations ranged from provider to societal. The studies pursuing a provider perspective tended to use data from non-governmental organizations, whereas the healthcare system perspective was used when the intervention compared a national programme with a supplemental programme. Only three studies utilized a societal perspective. Three studies did not specify the perspective, and it was assumed that these studies were likely from a provider perspective based on the use of administrative cost data. According to the World Health Organization (WHO) CHOICE guidelines on general cost-effectiveness analysis, a wider societal perspective is recommended to demonstrate all costs regardless of who pays (World Health Organization, 2003). The studies may not have taken a societal perspective because of the availability or difficulty of collecting cost data. This could be improved by planning a costing structure at the start of the study. Although the study perspective may not be societal, it is still important to make an inventory list of productivity costs of crisis-affected populations. This will be helpful for conducting sensitivity analysis to determine the strength of the parameters and assumptions. The information is also important to understand costs incurred by both providers and beneficiaries.

The higher quality studies were able to clearly state and justify the necessary components such as the time horizon, the analytical approach, discount rate and parameters. The studies also converted prices to appropriate currencies, accounted for inflation in the cost analysis, and presented the major outcomes in both aggregated and disaggregated forms. The weaker studies lacked a clear justification for conducting an economic study. These studies seemed to focus primarily on the effectiveness data and contained little information on the source of the cost data. This possibly indicates that the cost component was added later rather than at the inception of the project.

The weaker studies also used financial costs without converting them into economic costs, which include opportunity cost. In humanitarian setting, many programmatic interventions are supported by donations, which do not have any immediate financial consequence but there is an opportunity for costs of benefits forgone from the next best alternative use of those resources. The costs tended to be aggregated rather than evaluated on a unit basis. The studies mostly focused on shorter time horizons due to the humanitarian settings; although a proportion of fixed costs should have been apportioned to interventions, it often was not clear if and how this was done, and most studies included only variable costs instead. The reason for using variable costs might be because it was easier to estimate based on the quantity of inputs and outputs reported for the programme. Many studies also did not specify the discount rates for both costs and outcomes. One study reported a 10% discount rate, but this may be overestimated based on the inflation data from the conflict-affected country.

There also seemed to be problems in how outcomes were reported. The studies appeared to conduct cost-effectiveness by comparing the ratio of effectiveness measures over the total costs, rather than an incremental analysis (a ratio of the difference in costs and effects of the two alternatives). The reported outcomes tended to be in terms of cost per DALY averted, consistent with the WHO CHOICE guidelines that recommend the use of DALYs as an outcome measure (World Health Organization, 2003).

Another key aspect of economic evaluation is conducting a sensitivity analysis to test the robustness of the parameters and assumptions of the analytical model. The sensitivity analysis informs the readers whether the results will still be within a sizeable range if the parameters were shifted. Many of the studies did not report on a sensitivity analysis. Although some studies only included a one-way analysis, a robust approach for high-quality studies would include a probabilistic analysis.

Policy implications

There are clearly numerous challenges for incorporating economics in humanitarian programmes. First, substantial security and logistical challenges arise in the complex contexts of humanitarian crises, including population mobility and lack of control groups Humanitarian interventions must be prompt and so may allow little time for preparation for economic evaluations. There may also be capacity challenges, with humanitarian agencies lacking technical capacity to conduct such studies, including not having the robust information systems required to support economic evaluations (and more effective decision-making more broadly). There are also ethical constraints. For example, if the evaluation implies withholding assistance.

Despite the logistical, capacity and ethical challenges, many studies on the effectiveness of health interventions, including randomized controlled trial designs, have been conducted in such settings (Blanchet et al., 2017). Therefore, these challenges can be overcome in some circumstances, particularly more stable contexts such as longer-term refugee situations. Depending on the economic perspective used, data on costs can be relatively easily incorporated into effectiveness studies (although the economic costs, including opportunity costs, would need to be calculated). Economic evaluations reporting cost-effectiveness can then allow decision-makers to rank alternatives and provide a common measure to assess budgetary impact and identify programmes to scale up. The benefits could be substantial through improved efficiency, value for money, health benefits, equity and transparency and accountability. Indeed, Carbonnier (2016) noted the untapped potential for humanitarian economics.

There are a number of potential explanations for this gap in the use of economic evaluation studies. First, there may be limited awareness of the benefits of economic evaluation among donors, governments and agencies working on humanitarian response. This may be coupled with a lack of expertise on health economics in humanitarian responses and research. Second, there may also be misunderstandings around the role of economic evaluation, with a misconception that it may lead to withholding humanitarian assistance to particular vulnerable groups. This is not the case, and instead economic evaluations can help maximize benefits and improve equity. Third, there may also be a lack of political will, with donors and implementing agencies wary of conducting (cost)-effectiveness studies in case the outcomes are unfavourable (and instead preferring to focus on inputs and outputs). This calls for better transparency, accountability and governance in humanitarian organizations and the sector more broadly, including collecting and reporting data on economic evaluations of their activities (Blanchet et al., 2017; Roberts, 2018).

Clearly, not every programme’s cost-effectiveness should be evaluated; rather, cost-effectiveness should be established for key new interventions with limited or no existing cost-effectiveness data. There are also data from economic evaluations from interventions in more stable settings that could be applied or adjusted to humanitarian settings. Similarly, the timing of implementing cost-effectiveness studies is important for determining the use of economic evaluation methods. For example, although cost-effectiveness evidence for key sexual and reproductive health interventions is urgently needed (Singh et al., 2018), there is an argument for not evaluating the cost-effectiveness of implementing each sexual and reproductive health intervention at the onset of a crisis when costs will likely be higher than in a protracted setting.

Opportunities for economic evaluation

The inherent contextual constraints of conducting economic evaluations in humanitarian settings are fully recognized, but there is still an opportunity to conduct the (cost)-effectiveness studies to support policy-makers and funding institutions in decision-making (Ager et al., 2014). When effectiveness studies are being designed, it is important to also think about the cost component. The costing structure or economic evaluation can be integrated in the studies at the inception of the project rather than as an afterthought. Ensuring that accurate costing data are collected also requires sufficiently robust information systems. Study designs such as quasi-experimental designs, interrupted time series and/or matching can be used to help obtain the effectiveness data required for economic evaluations in the challenging contexts of humanitarian crises. Econometric methods can then be used to support the analysis. This information will also prove valuable in gaining new insights into the different types of costs associated with the intervention, including the productivity and opportunity costs, which can also help to inform future interventions. The perspective of the economic evaluation is crucial to determine and report, and where feasible and appropriate, a societal perspective should ideally be applied, to ensure that the full range of costs and benefits, particularly related to the recipients, is included.

Government, donors and humanitarian agencies and organizations need to start addressing the gap in using economic evaluation methods by better utilizing economists to understand and report on the economic challenges. They can also strengthen the capacity of policy-making staff to better use economic evaluation methods. Universities should strengthen teaching on the particular challenges of conducting economic evaluations in humanitarian settings. Investment is also required by donors and humanitarian agencies in improved information systems to allow for the better collection of more accurate, transparent, comparable and timely data for economic evaluation.

Study limitations

This review included only English-language studies, although articles in other languages may be relevant. In addition, this review only considered peer-reviewed articles and the inclusion of grey literature, expert interviews or reviews of references may have resulted in a greater number of eligible studies. Disease outbreak and epidemic search terms were not included in the search strategy as this would have significantly reduced the specificity of the search. Any epidemics or outbreaks of major importance are typically labelled as humanitarian emergencies by authorities and researchers/publications. This approach was based on expert consultation for other systematic reviews we have previously conducted on humanitarian crises (Blanchet et al., 2015). The quality scores were calculated by giving equal weightage for each question, which may skew the quality scores as it could be argued that some questions should have received more weight than others. This method for quality scores was chosen because the use of other published quality scores was not recommended by the CRD guidance (Centre for Reviews and Dissemination, University of York, 2009). This review only focused on full economic evaluations, while briefly highlighting the partial economic evaluations, thus limiting the total number of studies that could be identified for quality assessment. It is important to note that both full and partial economic evaluations are important in humanitarian settings, and further research could examine the quality of costing studies in humanitarian settings. The geographical location of the studies reported in this systematic review is mostly located in Africa. Thus, there is limited generalizability from individual study findings to other continents. This is particularly so given that humanitarian crises are very context specific so even more challenging to generalize than for other economic evaluations conducted in more stable settings (for which, even then, generalizability can be difficult).

Conclusion

Economic evaluations provide information on efficiency and, thus, are a good tool to use when assessing value for money in resource constraint settings. This review identified only 11 economic evaluations of public health interventions in humanitarian settings, LMICs, with most in sub-Saharan Africa. Areas for improvement in future economic evaluations were identified, including the sourcing of cost data, productivity changes, sensitivity analysis and valuation of health states and costs to improve the quality of economic studies. Furthermore, utilizing quasi-experimental techniques will aid in enhancing micro- and macroeconomic understanding of health interventions in humanitarian settings. The greater use of, and capacity in, economic evaluation methods in humanitarian crises would support decision-making, improve efficiency and value for money, prioritize and scale up health programmes and strengthen the transparency and accountability of humanitarian aid.

Acknowledgements

Salary support for this review was provided under the RECAP project by the United Kingdom Research and Innovation as part of the Global Challenges Research Fund (ES/P010873/1).

Conflict of interest statement. None declared.

Ethical approval. The Research Governance and Integrity Office of the London School of Hygiene and Tropical Medicine assessed the study protocol and determined that an ethics approval was not required for a systematic review.