Exploiting the emergent nature of mixed methods designs: insights from a mixed methods impact evaluation in Malawi

The application of mixed methods in Health Policy and Systems Research (HPSR) has expanded remarkably. Nevertheless, a recent review has highlighted how many mixed methods studies do not conceptualize the quantitative and the qualitative component as part of a single research effort, failing to make use of integrated approaches to data collection and analysis. More specifically, current mixed methods studies rarely rely on emergent designs as a specific feature of this methodological approach. In our work, we postulate that explicitly acknowledging the emergent nature of mixed methods research by building on a continuous exchange between quantitative and qualitative strains of data collection and analysis leads to a richer and more informative application in the field of HPSR. We illustrate our point by reflecting on our own experience conducting the mixed methods impact evaluation of a complex health system intervention in Malawi, the Results Based Financing for Maternal and Newborn Health Initiative. We describe how in the light of a contradiction between the initial set of quantitative and qualitative findings, we modified our design multiple times to include additional sources of quantitative and qualitative data and analytical approaches. To find an answer to the initial riddle, we made use of household survey data, routine health facility data, and multiple rounds of interviews with both healthcare workers and service users. We highlight what contextual factors made it possible for us to maintain the high level of methodological flexibility that ultimately allowed us to solve the riddle. This process of constant reiteration between quantitative and qualitative data allowed us to provide policymakers with a more credible and comprehensive picture of what dynamics the intervention had triggered and with what effects, in a way that we would have never been able to do had we kept faithful to our original mixed methods design.


Introduction
In recent years, the application of mixed methods in Health Policy and Systems Research (HPSR) has expanded remarkably. A scoping review published in 2018 focused specifically on sub-Saharan Africa and, tracing publications as far back as 1950, shows an exponential increase over time in the number of publications defined as mixed methods (De Allegri et al., 2018). The same review, however, also pointed at substantial methodological weaknesses in the existing literature. In particular, the review highlighted how only about half of all publications defined an explicit mixed methods research question and less than one-fifth employed a clearly outlined mixed methods design. Most studies applied quantitative and qualitative methods, but did not necessarily conceptualize the two as part of a single research effort, hence failing to make full use of integrated approaches to data collection and analysis. In particular, the review revealed that very few studies exploited the emergent nature of mixed methods designs as a distinctive feature of this research approach (Creswell and Plano Clark, 2011). Most publications reported that quantitative and qualitative data were collected and analysed concomitantly or sequentially, with little information flow from one strain of work to the other in an iterative process aimed at shaping further data collection and analysis.
In this article, we wish to illustrate how explicitly acknowledging the emergent nature of mixed methods designs can lead to a richer and more informative application in HPSR. Instead of illustrating our point by reiterating the existing literature on mixed methods (Ozawa and Pongpirul, 2014), we do so by reflecting on our experience working on the evaluation of a complex health system intervention, specifically the Results Based Financing for Maternal and Neonatal Health (RBF4MNH) initiative in Malawi (Brenner et al., 2014).

Defining mixed methods research
Before illustrating our point through the description of our own work in Malawi, we outline our understanding and operational definition of mixed methods research.
The multiple definitions employed in the literature converge to recognize mixed methods research as the combination of quantitative and qualitative methods of data collection and analysis within a single research effort (Tashakkori and Teddlie, 2010;Creswell and Plano Clark, 2011;Ozawa and Pongpirul, 2014;De Allegri et al., 2018). We recognize mixed methods research as a distinctive research approach, which emerged in the early 21st century (Ö stlund et al., 2011;Pluye and Hong, 2014), moving beyond earlier attempts to integrate quantitative and qualitative methods, but doing so outside the framework of an explicit research paradigm accommodating the two (Pluye, 2012). As mixed methods researchers, we make an explicit effort to move beyond either positivism or social constructivism to build on an epistemology of pragmatism (Johnson and Onwuegbuzie, 2004). For our research, this means that rather than being driven by a priori preference for a given approach, we let the research question guide the choice of the methodology to be applied in our work (Creswell and Plano Clark, 2011). Borrowing directly from Greene's (2007Greene's ( , 2008 language, we further acknowledge that as mixed methods researchers, we are called to let multiple mental models co-exist within a single research effort and that by doing so we legitimize multiple approaches to social inquiry. Moreover, we position our work as a multi-method approach to mixed methods. In line with existing theoretical literature (Tashakkori and Teddlie, 2010;Creswell and Plano Clark, 2011), we refer to our work as multi-method to indicate adherence to a single research paradigm within the individual quantitative and/or qualitative research strains of a mixed methods study. In practice, this means that we do not push the epistemological boundaries of the single research strains, so that we keep truthful to the underlying assumptions that pertain specifically to either quantitative or qualitative research.
We view the emergent nature of mixed methods designs as a key feature of this methodological approach. We define as emergent any design that is born out of an iterative dialogue between quantitative and qualitative strains of data collection and analysis, allowing for revisions of sampling, data collection and data analysis strategies as a study unravels. The case below illustrates how working with emergent designs represents a key comparative advantage of mixed methods research, offering a unique opportunity for more meaningful and credible knowledge generation in HPSR. Notwithstanding this statement, we recognize that not all mixed methods studies ought to rely on emergent designs and that pre-set designs sometimes represent the most suitable methodological approach.

Understanding shifts in demand for obstetric services in Malawi
The case we use to illustrate our point reflects our experience while conducting the impact evaluation of the RBF4MNH initiative, an intervention implemented in four rural districts in Malawi between 2013 and 2018. The RBF4MNH initiative combined performancebased payments to maternal care providers with conditional cash transfers to pregnant women to improve access to quality obstetric services. The impact evaluation as a whole adopted a mixed methods design, described elsewhere (Brenner et al., 2014). Hereafter, we focus exclusively on how we relied on an emergent mixed methods design to understand a specific set of findings, which arose unexpectedly half way through the impact evaluation. Figure 1 provides a simplified illustration of the steps undertaken, allowing the reader to locate the actions described below in time.
One of the primary objectives of the impact evaluation was to assess the effect of the RBF4MNH initiative on women's utilization of obstetric services. Towards this aim, we had set up a quantitative household survey on a sample of approximately 2000 households to monitor health service utilization across intervention and control Key Messages • Mixed methods studies are increasing in the field of health policy and health system research.
• Most mixed methods studies still make little use of emergent designs, a key feature of this methodological approach. • We illustrate the potential of mixed methods studies to make better use of emergent design by reflecting on our own experience conducing a mixed methods impact evaluation in Malawi. • Our experience suggests that relying on an emergent design, integrating multiple sources of quantitative and qualitative data, can generate more credible evidence for policy.
facilities. At mid-term, standard difference-in-differences (DID) analysis (Angrist and Pischke, 2009) of survey data indicated that the intervention had not resulted in increased utilization of maternal care services among women residing in intervention compared with control areas (Brenner et al., 2018). We judged this finding to be plausible given that at baseline we had already detected higher-thanexpected utilization rates across intervention and control areas. Hence, we were aware that our study might not be powered to detect changes of small magnitude, should those have even occurred. Parallel to the household survey, we had also conducted qualitative interviews both with healthcare workers and service users, with the aim of exploring experiences and unravelling mechanisms responsible for change (or lack thereof) attributable to RBF4MNH. Interviews with women revealed no information that challenged our survey findings (Kambala et al., 2017), whereas interviews with health workers were troubling as they contradicted the view that the intervention had not produced an increase in utilization of maternal healthcare services in intervention areas (Lohmann et al., 2018a). Specifically, health workers in intervention facilities consistently complained of increased workload due to increased demand for maternal care services, which they attributed to RBF4MNH because no other intervention was ongoing in the four districts.
We discussed the matter repeatedly, but after checking the analysis of the household survey multiple times, we convened to trust the quantitative findings. We initially attributed health workers' accounts of increased workload to time-consuming quality improvements or simply to their perceptions rather than to an actual increase in service use. Still, not being fully convinced of simply disregarding the qualitative findings, we actively sought additional sources of information, with the intention of being able to provide evidence in favour of either one of the two emerging realities.
To do so, we turned to a different source of quantitative data and decided to take a second look at the same issue, e.g. utilization of maternal care services, but to shift the perspective from that of communities using services to that of providers delivering them. We adjusted our study design to include an additional quantitative component and performed an interrupted time series analysis (ITSA) with independent controls using routine health facility data (Shadish et al., 2002;Lagarde, 2012). We knew that these data were of sufficient quality to be used for scientific assessments and that they were independent of reporting for the RBF4MNH initiative. The ITSA revealed a sharp increase in the number of deliveries and antenatal care (ANC) visits in intervention compared with control facilities following the RBF4MNH launch. Now facing two conflicting realities within the quantitative study component, one emerging from survey data and one from routine health facility data, we decided to return to the household survey data to trace specifically in which facilities women in our sample had delivered. This more detailed analysis of household survey data uncovered something that our initial aggregated DID analysis had overlooked: 1 year after the intervention launch, the proportion of women moving from control to intervention facilities for delivery services had grown by 9 percentage points, from 16% at baseline to 25 at mid-term . This increased migration from control to intervention facilities represented a substantial challenge for our impact evaluation, because it signalled contamination and hence made effect identification difficult (Shadish et al., 2002;Angrist and Pischke, 2009). At the same time, the presence of this contamination legitimized the health workers' statements on workload gathered during the qualitative interview. In addition, this finding triggered our curiosity to dig deeper into understanding sources of contamination, so we adjusted our design once again and returned to the field for more qualitative interviews.
More specifically, we returned to the field to understand what had motivated this substantial increase in migration from control to intervention facilities. Our hypothesis was that women had redirected themselves from control to intervention facilities under the false assumption of being eligible for the cash transfers. To explore our hypothesis, we interviewed specifically those women who had migrated from control to intervention facilities for their delivery. The availability of the household survey data made tracing these women possible. Surprisingly, all the women we interviewed unanimously reported being referred to deliver in an intervention facility by the health workers employed at their local facilities . Contrary to our hypothesis, all interviewed women knew that the cash transfers were limited to the women residing in intervention areas, so they had no expectation of making a financial gain by delivering outside of their catchment area. Rather the opposite, they were willing to incur additional costs due to transport to comply with their health workers' referral, because they were convinced that he could better judge where they should deliver their baby. In addition, women revealed frequently moving to an intervention facility already for their later ANC visits (to be known to the health workers), thus explaining also the increase in ANC visits detected by the ITSA.
Parallel interviews with health workers in control facilities confirmed women's narratives, while also offering additional information on the matter . Listening to women's recall, we had come to believe that health workers at control facilities had simply encouraged women at higher risk of complications to visit intervention facilities, trusting that those facilities would have better resources to intervene in case of emergency. Talking to health workers, however, suggested that a large portion of the increased migration from control to intervention facilities was actually due to the fact that all four public secondary level facilities in the four target districts had been selected as intervention facilities. Hence, increasing referral rates towards intervention facilities were largely linked to increased referral rates from control first level towards intervention secondary level (i.e. from Basic Emergency Obstetric Care (BEmOC) to Comprehensive Emergency Obstetric Care (CEmOC) facilities).
To confirm the hypothesis emerging from this latest round of qualitative interviews, we returned once again to our quantitative data and using a mixture of household survey, primary health facility data (also collected as part of our study) and routine health facility data, we explored the evolution of referral patterns in the four study districts. Although the data were far from being sufficiently complete to allow the in-depth analysis we would have liked to conduct, we were able to identify distinctly different referral patterns across facilities, with control facilities having increased and intervention facilities having decreased referrals to the district CEmOC facilities following the RBF4MNH launch. We checked plausibility of this observation with RBF4MNH stakeholders and they confirmed that reducing referrals from BEmOC to CEmOC facilities was an intended objective of their programme (albeit not one that was linked to a specific financial incentive), the aim being that of empowering and enabling lower-level facilities to adequately handle more births in order to curb overcrowding of maternity wards in district hospitals. In designing the programme, however, neither the government not its development partners had accounted for the unintended consequence of indirectly motivating referrals to CEmOC facilities from health workers stationed at control BEmOC facilities.
What we learned along the way?
We trust that our experience provides a useful illustration of the value and feasibility of relying on emerging designs when conducting mixed methods research. Had we stuck to the original study design as described in the protocol (Brenner et al., 2014) and not sought to investigate the initial discrepancies between quantitative and qualitative findings, we would have likely falsely concluded that that the intervention had produced no effects on demand for obstetric services. We followed up on the initial discrepancy in a step-wise manner, making use of multiple data sources and multiple analytical methods within both the quantitative and the qualitative study components. This allowed us to understand that the intervention had not increased utilization rates at the population level, but had nevertheless produced important shifts in demand, with an increasing number of women migrating from control to intervention facilities. Furthermore, contrary to our expectations, we learned that this shift was largely driven by the supply side and linked to unexpected changes in referral practices among control healthcare providers. As a research team, we are concerned that had we not adjusted our design multiple times and made use of a multi-method approach within the broader mixed methods study, we would have ended up disseminating inaccurate and incomplete findings and hence providing misleading policy advice. The description of our step-wise investigation into understanding shifts in demand for obstetric services largely conceals the challenges we actually faced when conducting the work. Adjusting the study design multiple times required adjusting the work of a large team of people engaged in the overall impact evaluation. These adjustments were especially challenging for the more junior researchers working on the project and for those in charge of fieldwork. Not only did these adjustments entail delays in overall scientific production (because we held onto data and delayed any dissemination until we were sure to understand exactly what had happened), but often challenged the position of the individual researchers, because not all of them were equally versed in mixed and multi-methods research at the onset of the study.
Nevertheless, reflecting back on our experience, we can identify four key success factors that allowed us to follow an emergent study design to accommodate emergent knowledge needs. First, time played to our favour, because we had 3 years (later extended to four) to complete our work. Hence, we could afford the privilege of moving back and forth between sources of data and emerging findings in a way that shorter-term research consultants would never have. Second, we held frequent meetings (weekly or biweekly during the most intensive project phases) bringing together the entire team to discuss findings emerging across strains of analysis. Our impact evaluation was extremely broad and alongside investigating the effects of the RBF4MNH initiative on utilization of obstetric services, we also investigated its effects on health workers' motivation and quality of service delivery, as well as the intervention's acceptability, adoption and adaptation at multiple levels of implementation. To cover such a broad range of issues, our team was very diverse, including qualitative social scientists, clinicians, epidemiologists, psychologists and health economists. These meetings were essential to ensure a constant dialogue, enabling the single researchers to see beyond their own epistemological and methodological convictions, pushing the work itself towards a continuous progressive iteration and integration across quantitative and qualitative elements. Third, we need to acknowledge that the team comprised only researchers who explicitly committed to work within a mixed methods framework, valuing the contribution of qualitative methods to impact evaluation. As noted earlier, actual expertise in mixed and multi-methods research varied substantially across team members, but all researchers working on the project had been informed upon recruitment of the Principal Investigator's methodological orientation and attended at least the 2-week advanced training in mixed methods research in international health offered at the Heidelberg Institute of Global Health. Last but surely not least, we relied on the support of very flexible funding agencies. In particular, we must acknowledge the support received by the intermediary agency administering our grant not only in discussing critically with us the proposed changes but also in accommodating these changes contractually multiple times over the course of the project lifetime. Likewise, the development partners directly engaged in Malawi agreed to multiple changes to the field budget and extended the fellowships of the students engaged on the project by a 6-month period, so that we could gain time and be true to our research aim by adjusting our initial fairly static design.
While we trust that the adjustments made were adequate to address emerging challenges, we do not assert that our methodological choices represented the only viable strategy to reconcile the apparent discrepancy between quantitative and qualitative findings. We do recognize that a different team with a different set of expertise might have chosen to follow a different design, investing in a different set of additional data collection and analysis activities. Similarly, one could argue that a different decision on study design could have been made a priori, because contamination was to be expected if drawing intervention and control facilities from the same districts. As described in detail in our prior work, however (Brenner et al., 2014(Brenner et al., , 2017(Brenner et al., , 2018Kambala et al., 2017;Lohmann et al., 2018b), our initial design was constrained by a broader set of considerations related to the implementation of the RBF4MNH initiative as well as to the parallel implementation of another PBF scheme, the Support for Service Delivery Integration Performance-Based Incentives (SSDI-PBI) (McMahon et al., 2016). In this article, however, we do not argue that we adjusted our design to account for contamination, but rather that we exploited the initial indication that some contamination had actually taken place to understand the real-life dynamics triggered by the RBF4MNH initiative.

Conclusion
We trust that our description of how we adjusted our study design multiple times in response to the evidence arising from either the quantitative or the qualitative study component underlines the value and feasibility of relying on emerging designs when conducting mixed methods research. Had we stuck to the original study design and not sought to investigate discrepancies between quantitative and qualitative findings, we would have likely falsely concluded that that the intervention had produced no effects on demand for obstetric services, hence providing misleading policy advice. Constant dialogue within our multi-disciplinary research team, good understanding of the principals of mixed methods research among all research team members, ability to access the necessary data and/ or to invest in additional primary data collection and flexibility on the part of funding agencies emerged as important success factors. By highlighting the need to integrate qualitative research in impact evaluation studies, our experience clearly points at the advantages of employing mixed methods when confronted with the challenge of analysing the effects of complex health system interventions. It is, therefore, our hope that an increasing number of researchers will turn to them to find responses to puzzling and complex questions we face in HPSR.