- Split View
-
Views
-
Cite
Cite
M De Allegri, I Sieleunou, G A Abiiro, V Ridde, How far is mixed methods research in the field of health policy and systems in Africa? A scoping review, Health Policy and Planning, Volume 33, Issue 3, April 2018, Pages 445–455, https://doi.org/10.1093/heapol/czx182
- Share Icon Share
Abstract
Both the academic and the policy community are calling for wider application of mixed methods research, suggesting that combined use of quantitative and qualitative methods is most suitable to assess and understand the complexities of health interventions. In spite of recent growth in mixed methods studies, limited efforts have been directed towards appraising and synthetizing to what extent and how mixed methods have been applied specifically to Health Policy and Systems Research (HPSR) in low- and middle-income countries (LMICs). We aimed at filling this gap in knowledge, by exploring the scope and quality of mixed methods research in the African context. We conducted a scoping review applying the framework developed by Arksey and O’Malley and modified by Levac et al. to identify and extract data from relevant studies published between 1950 and 2013. We limited our search to peer-reviewed HPSR publications in English, which combined at least one qualitative and one quantitative method and focused on Africa. Among the 105 studies that were retained for data extraction, over 60% were published after 2010. Nearly 50% of all studies addressed topics relevant to Health Systems, while Health Policy and Health Outcomes studies accounted respectively for 40% and 10% of all publications. The quality of the application of mixed methods varied greatly across studies, with a relatively small proportion of studies stating clearly defined research questions and differentiating quantitative and qualitative elements, including sample sizes and analytical approaches. The methodological weaknesses observed could be linked to the paucity of specific training opportunities available to people interested in applying mixed methods to HPSR in LMICs as well as to the limitations on word limit, scope and peer-review processes at the journals levels. Increasing training opportunities and enhancing journal flexibility may result in more and better quality mixed methods publications.
A number of organizations are calling for wider application of mixed methods research in Health Policy and Systems Research (HPSR).
The last few years have been characterized by an increase in number of studies being published as mixed methods studies, within the field of HPSR in Africa.
Three journals (Health Policy and Planning, BMC Health Services Research and Social Science and Medicine) accounted for a fourth of all published HPSR mixed methods articles on Africa.
The suboptimal quality of the published mixed methods studies calls for strengthening capacity in mixed methods research, especially for the health sciences in low- and middle-income countries.
Introduction
In 2012, the Alliance for Health Policy and Systems Research published a Methodological Reader (Gilson 2012), inviting researchers working in the field to rely more extensively on the use of mixed methods in their research practice. The publication of this document formalized previous statements by the World Health Organization in favour of mixed methods as a new paradigm for health policy and systems research (HPSR) (World Health Organization 2012). Subsequently, mixed methods gained increased attention in the literature (Gilson et al. 2011) to the point that leading research institutions, including the National Institutes of Health (NIH) in the United States (Creswell et al. 2011) and the Medical Research Council (MRC) in the United Kingdom (Moore et al. 2015), recently issued documents encouraging the application and offering guidance on the development of rigorous mixed methods protocols. Still, from the information we could retrieve, funding devoted to mixed methods studies appears to remain modest. For instance, public information available indicates that <3% of all Canadian Institute for Health Research funding was assigned to mixed methods studies between 2009 and 2012. Other funding agencies, including for example National Institutes for Health in the United States or the Germa Research Society (DFG), do not differentiate funding assigned by methodological approach, but only by thematic area, making it impossible to estimate to what extent the claims in favour of mixed methods are met by actual funding decisions.
This series of public statements in favour of mixed methods research emerged amidst a research context already widely contaminated by the application of mixed methods approaches. In the literature, mixed methods research is described using a diverse range of definitions (Ozawa and Pongpirul 2014). All the existing definitions, however, converge to recognize mixed methods research as the combination of quantitative and qualitative methods of data collection and analysis within a single research effort with the aim of counteracting the weaknesses and building on the strengths of the single quantitative and qualitative approaches (Tashakkori and Teddlie 2010; Creswell and Plano Clark 2011). Although early attempts to combine quantitative and qualitative methods of data collection and analysis in global health were already documented in the 1950s (Pluye 2012), the concept of mixed methods as a distinct and explicit approach to research only emerged at the beginning of the 21st century (Östlund et al. 2011; Pluye and Hong 2014). It has been argued that mixed methods research is particularly suitable to research in health policy and systems given the complexity of the policies, systems, interventions and contexts that researchers in this field are called to document (Clarke and Yaros 1988; O’Cathain 2009). Furthermore, it has been widely recognized that the increased interest in mixed methods research in health policy and systems is largely attributable to a pragmatic concern, as researchers are urged to produce evidence that adequately inform policy beyond the epistemological debates between positivists and social constructivists that have long dominated scientific inquiry (O’Cathain et al. 2007; Gilson 2012; Pluye and Hong 2014; Saetren 2014; Bishop 2015; Robert and Ridde 2016; Sheikh et al. 2016).
Limited efforts, however, have been directed towards collectively appraising and synthetizing mixed methods studies in health, especially in relation to their application in low- and middle-income countries (LMICs). Pluye and colleagues have led the development of a Mixed Methods Appraisal Tool (Souto et al. 2015) to guide the conduct and publication of mixed methods reviews, aimed at synthetizing evidence on a specific health issue or a health intervention (Pluye et al. 2009). Their work has built on earlier efforts by O’Cathain et al. (2008) to develop a framework to assess the quality of mixed methods studies. The work of Pluye’s team has been fundamental in advancing both the conceptualization and the practice of mixed methods reviews, with applications across settings (Pluye et al. 2016), but has not led to a synthesis of the existing body of research adopting a mixed methods approach. Two reviews have been conducted to appraise state of practice in mixed methods research in the health field, one aimed at assessing quality in mixed methods research (O’Cathain et al. 2008) and one aimed at exploring and synthetizing existing analytical approaches in mixed methods research (Östlund et al. 2011). Both reviews, however, were largely based on studies that had originated in high-income countries, with no attention being paid to the practice of mixed methods research in LMICs.
We aimed to fill this existing gap in knowledge by conducting a scoping review of mixed methods studies specific to the field of HPSR and focused on Africa. In line with the theoretical foundation of the methodological approach adopted (Arksey and O’Malley 2005; Levac et al. 2010), our scoping review aims at providing a comprehensive mapping of the practice of mixed methods research in Africa. Our mapping includes an assessment of quality as a means of appraising how mixed methods studies are carried out in Africa. The geographical focus on Africa is justified by our own expertise, as four scientists engaged in mixed methods research in this continent. We felt that we could best appraise studies if we limited the scope of our review to a continent we know well.
Methods
We conducted our scoping review applying the framework developed by Arksey and O’Malley (2005), with the modifications suggested by Levac et al. (2010). In line with this approach, hereafter, we describe: (1) the definition of the research question; (2) the identification of the relevant studies; (3) the selection of the studies; (4) the data charting; and (5) the collating, summarizing and reporting of the results.
The definition of the research question
Our study sought to answer the following research question: ‘What is the state of use of mixed methods approaches in HPSR in Africa?’ We wanted to understand the extent of the application of mixed methods in HPSR in Africa, the specific nature of the mix of methods being used, as well as the countries, the fields of interests and the contexts within which such a methodological approach was applied. Although we did not aim at judging the strength of the evidence emerging from the single studies, we did appraise quality of the mixed methods studies reviewed by looking specifically at: (1) whether a clear mixed methods research question had been asked; (2) whether a clear mixed-methods design had been described; (3) whether the specific quantitative and qualitative methods applied were suitable to answer the research question, that is to say whether the methods applied were clearly aligned with the research question; and (4) whether findings had been integrated and discussed, that is to say whether the emerging interpretation was based on a joint appraisal of quantitative and qualitative findings.
For operational purposes, we defined mixed methods research as research that encompasses a variety of approaches which rest on the integration of quantitative and qualitative methods of data collection and analysis (Pluye and Hong 2014). Specifically, we defined as mixed methods any study which integrated at least one of the following: (1) qualitative and quantitative research questions; (2) qualitative and quantitative research designs; (3) quantitative and qualitative techniques for collecting and analysing data; and (4) qualitative and quantitative results. We further defined HPSR as research ‘that seeks to understand and improve how societies organize themselves to achieve collective health goals, and how different actors interact in the policy and implementation processes to contribute towards policy outcomes’ (original statement by the Alliance on Health Policy and Systems research, cited by Gilson in the Health Policy and Systems Research Methodological Reader, page 21). HPSR is by nature interdisciplinary and explores the link between heath policies, health systems, health outcomes and broader determinants of health (Gilson 2012).
The identification of relevant studies
In our search strategy, we aimed to be as comprehensive as possible in identifying primary studies and reviews published in peer-review journals. To achieve this, we searched for research evidence via electronic databases. We started with the ambition of including also evidence from grey literature, but once we began the review, we realized that all grey literature initially identified did not qualify for inclusion in the review.
With the assistance of a librarian, the second author initiated a comprehensive search of bibliographic databases (with citation tracking) and the third author validated the search. We used three main databases for this purpose: Pubmed, Embase and Cochrane. To identify additional articles in peer-review journals, we also searched Google Scholar and Social Care databases. We did not search databases of research organizations and/or other internet sources, because, as mentioned above, we limited our search to primary studies and reviews published in peer-review journals.
We used the following search terms related with methods, content and location: ‘mixed methods’, ‘mixed methods review’, ‘qualitative and quantitative’, ‘multi-method’, ‘multiple methods’, ‘mixed methods approach’, ‘health systems’, ‘health policy’, ‘health policy analysis’, ‘health system development’, ‘health system strengthening’, all combined with both ‘Africa’ and ‘sub-Saharan Africa’ as well as with the name of each country in the continent (e.g. ‘Angola’, ‘Mozambique’, ‘Burkina Faso’, ‘Kenya’, etc.).
Eligibility criteria
We included all articles: (1) that combined at least one qualitative method (QUAL) and one quantitative method (QUAN) in any of the four dimensions outlined earlier (research questions, study design, data collection and results/analysis); (2) focused on Africa or one of its countries; and (3) were pertinent to HPSR. As outlined in the section above, our operational definition of mixed methods is rather broad. As such, our search strategy allowed for the inclusion of studies that did not strictly comply with a purist mixed methods design, but would fall within the category that Creswell and Plano define as gray area mixed methods studies (Creswell and Plano Clark 2011). We considered studies eligible for the scoping review as long as they combined quantitative and qualitative methods, irrespective of the specific contribution made by either method. To limit the scope of the review and to ensure alignment with HPSR principles, we further restricted the review to articles addressing system-level issues and interventions (such as health care financing, health service delivery and human resources for health) and to exclude articles focused on disease-specific control programs and interventions (such as malaria, tuberculosis and human immunodeficiency virus/acquired immune deficiency syndrome).
Time span
We searched for articles published between 1950 and December 2013. We chose this time period because, according to Pluye et al. (2012), the combination of quantitative and qualitative methods was first introduced in global health research in the 1950s.
Language
We restricted our search to studies published in English. We excluded other languages due to cost and time limitations and in light of the consideration that the literature published in English would amply cover the existing body of evidence. This does not mean that we do not recognize the value of literature published in other languages, but that, due to its central role in local academic discussions, the English literature would be sufficient to unravel patterns in relation to our research question.
At the identification stage, we retrieved a total of 1454 articles (Pubmed = 747 articles, Embase = 643 articles, Cochrane = 64 articles), of which 1218 remained after removing duplicates.
The selection of the studies
The second and the third author concomitantly reviewed the 1218 abstracts. Their work unraveled along two stages.
The first screening stage focused on identifying all articles that met the inclusion criteria. Inter-selection agreement between the two authors was 99.2%. Consensus on the remaining ten articles (0.8%) was reached through consultation with the two senior authors. This first stage led to the retention of 173 studies to be included in the review.
We could not retrieve the full-text for 16 out of the 173 abstracts included in the first stage of screening (5 conference abstracts, 1 poster and 10 full-texts not available—due to publication in minor African journals that were not accessible through any of our library systems). After review of the full text, we excluded an additional 52 articles from the 157 retrieved. Reasons for exclusion included: 42 articles were not mixed-method; 3 focused on a specific disease; 6 were not done in Africa; and 1 was not in English. Finally, after this second screening stage, we retained 105 articles for data extraction (Figure 1).
Charting the data
We developed a data extraction form (in Excel) to classify information from the retained studies. In line with the objectives of a scoping review and with our research question, the data extraction form reflected the wish to describe both the scope and the quality of the published HPSR literature. For completeness of information, the data extraction form is included as Supplementary Appendix S1 to this article. Data extraction criteria were defined collectively ex ante by all authors. In particular, the framework developed by Pluye et al. (2009) was adjusted to extract information on the quality of the studies reviewed.
We worked as a team on the first 15 articles to ensure consistency in the extraction process and then the second and the third authors extracted data in parallel from an equal number of articles. The senior authors checked for consistency of the process by double-checking data extraction procedures on a number of randomly selected articles.
Collating, summarizing and reporting the results
We relied on a mixture of quantitative and qualitative approaches to synthetize findings. We used descriptive and frequency tables to aggregate the easy-to-list information extracted from the articles (e.g. publication year, regional focus), while we used a narrative approach to synthetize more complex information (e.g. integration of qualitative and quantitative findings, themes). To ease understanding by the reader, we created a number of categories to classify the information extracted. We distinguished three publication time periods, starting with the first year we could identify a relevant article, (1990–99; 2000–09; 2010–13) and five African regions (North, East, Central, West, Southern). To categorize articles according to their research focus, we used the classification proposed in HPSR (Gilson 2012) and distinguished articles across: Health Systems (Human Resources, Financing, Medicine and Equipment, Health Information, Service Delivery, Governance); Health Policy (Agenda Setting, Decision Making, Implementation, Evaluation); and Health Outcomes (General Health Outcomes, Inequity, Service Utilization).
We worked as a team through the process of synthetizing findings, re-working the classification of the information several times in an iterative manner. In particular, the two senior authors took the responsibility to appraise the quality of the studies reviewed on the basis of the information on design, sampling, analysis and integration included in the data extraction form. To appraise quality, the two senior authors engaged in a process on interactive discussion, informed by the data extracted by the two junior authors along the specific criteria outlined in Table 1. The criteria used for quality appraisal were defined a priori by all authors collectively and are illustrated also in the definition of our research question.
. | Number of studies (N = 105) . | % . |
---|---|---|
Is the MMR design relevant to address the research question (or objective)? | ||
Yes | 72 | 68.6 |
No | 33 | 31.4 |
Is the data collected adequate to address the research question (or objective)? | ||
Yes | 69 | 65.7 |
No | 36 | 34.3 |
Are findings presented in an integrated or in a separate manner? | ||
Integrated manner | 65 | 61.9 |
Separate manner | 40 | 38.1 |
Are findings discussed in an integrated or in a separate manner? | ||
Integrated manner | 101 | 96.2 |
Separate manner | 4 | 3.8 |
Is the level of integration relevant to address the research question (objective)? | ||
Yes | 82 | 78.1 |
No | 23 | 21.9 |
Is appropriate consideration given to the limitations associated with this integration? | ||
Yes | 23 | 21.9 |
No | 82 | 78.1 |
Is appropriate consideration given to how findings relate to the context? | ||
Yes | 52 | 49.5 |
No | 53 | 50.5 |
Is appropriate consideration given to how findings relate to researchers’ influence? | ||
Yes | 10 | 9.5 |
No | 95 | 90.5 |
. | Number of studies (N = 105) . | % . |
---|---|---|
Is the MMR design relevant to address the research question (or objective)? | ||
Yes | 72 | 68.6 |
No | 33 | 31.4 |
Is the data collected adequate to address the research question (or objective)? | ||
Yes | 69 | 65.7 |
No | 36 | 34.3 |
Are findings presented in an integrated or in a separate manner? | ||
Integrated manner | 65 | 61.9 |
Separate manner | 40 | 38.1 |
Are findings discussed in an integrated or in a separate manner? | ||
Integrated manner | 101 | 96.2 |
Separate manner | 4 | 3.8 |
Is the level of integration relevant to address the research question (objective)? | ||
Yes | 82 | 78.1 |
No | 23 | 21.9 |
Is appropriate consideration given to the limitations associated with this integration? | ||
Yes | 23 | 21.9 |
No | 82 | 78.1 |
Is appropriate consideration given to how findings relate to the context? | ||
Yes | 52 | 49.5 |
No | 53 | 50.5 |
Is appropriate consideration given to how findings relate to researchers’ influence? | ||
Yes | 10 | 9.5 |
No | 95 | 90.5 |
. | Number of studies (N = 105) . | % . |
---|---|---|
Is the MMR design relevant to address the research question (or objective)? | ||
Yes | 72 | 68.6 |
No | 33 | 31.4 |
Is the data collected adequate to address the research question (or objective)? | ||
Yes | 69 | 65.7 |
No | 36 | 34.3 |
Are findings presented in an integrated or in a separate manner? | ||
Integrated manner | 65 | 61.9 |
Separate manner | 40 | 38.1 |
Are findings discussed in an integrated or in a separate manner? | ||
Integrated manner | 101 | 96.2 |
Separate manner | 4 | 3.8 |
Is the level of integration relevant to address the research question (objective)? | ||
Yes | 82 | 78.1 |
No | 23 | 21.9 |
Is appropriate consideration given to the limitations associated with this integration? | ||
Yes | 23 | 21.9 |
No | 82 | 78.1 |
Is appropriate consideration given to how findings relate to the context? | ||
Yes | 52 | 49.5 |
No | 53 | 50.5 |
Is appropriate consideration given to how findings relate to researchers’ influence? | ||
Yes | 10 | 9.5 |
No | 95 | 90.5 |
. | Number of studies (N = 105) . | % . |
---|---|---|
Is the MMR design relevant to address the research question (or objective)? | ||
Yes | 72 | 68.6 |
No | 33 | 31.4 |
Is the data collected adequate to address the research question (or objective)? | ||
Yes | 69 | 65.7 |
No | 36 | 34.3 |
Are findings presented in an integrated or in a separate manner? | ||
Integrated manner | 65 | 61.9 |
Separate manner | 40 | 38.1 |
Are findings discussed in an integrated or in a separate manner? | ||
Integrated manner | 101 | 96.2 |
Separate manner | 4 | 3.8 |
Is the level of integration relevant to address the research question (objective)? | ||
Yes | 82 | 78.1 |
No | 23 | 21.9 |
Is appropriate consideration given to the limitations associated with this integration? | ||
Yes | 23 | 21.9 |
No | 82 | 78.1 |
Is appropriate consideration given to how findings relate to the context? | ||
Yes | 52 | 49.5 |
No | 53 | 50.5 |
Is appropriate consideration given to how findings relate to researchers’ influence? | ||
Yes | 10 | 9.5 |
No | 95 | 90.5 |
Ideally, to ensure completeness of the process, we should have also sought out direct discussions with the broader network of African research stakeholders (including both researchers and research users). This was not feasible due to time and resource constraints. Still, as authors, we plan to seek out opportunities for this open discussion during upcoming HPSR conferences.
Ethical considerations
Given the review nature of our study, we did not need to obtain ethical clearance before performing the review. All studies included were published and as such publicly available.
Results
To ease reading, we report results across six sub-sections, each focused on a specific set of information contained in our data extraction form: Date of publication, journals and regional focus; Research focus and study population; Clarity of research question and study design; Quantitative study components; Qualitative study component; Integration across quantitative and qualitative findings. In the sections below, we illustrate good practice in mixed methods research using single studies as examples.
Date of publication, journals and regional focus
Figure 2 shows the distribution of articles by date of publication. Out of 105 reviewed articles, 61 (nearly 60%) were published in 2010 and afterwards, compared with only three (3%) being published in the 1990–99 decade. Nearly 60% of all articles were published in open access sources.
With seven to eight articles each, three journals (Health Policy and Planning, BMC Health Services Research and Social Science and Medicine) alone accounted for nearly a fourth of all published HPSR mixed methods articles on Africa. These three journals were followed by a number of other journals having published two to three (2–3%) of all retrieved HPSR mixed methods articles on Africa. These journals were either African-based journals (African Journal of Reproductive Health, East African Medical Journal and South African Family Practice) or reproductive health journals (BMC Pregnancy and Childbirth, and International Journal of Gynecology and Obstetrics). Only two non-disease and non-continent specific health journals (PlosOne and WHO Bulletin) appeared among the journals having published two or three mixed methods articles. Two additional journals (Nurse Education in Practice and Public Health Nutrition) had published two HPSR mixed methods articles. The remaining articles were scattered among 59 field-specific journals (e.g. Malaria Journal, Tropical Medicine and International Health and Studies in Family Planning), each having published one single mixed methods article. No mixed methods article relevant to HPSR was retrieved in a leading medical journal, such as The Lancet or The British Medical Journal (including sister journals).
With 37 out of 105 articles (35%), East African countries represented the main focus of published HPSR mixed methods research, followed by Southern Africa with 30 (29%) articles and by West Africa with 25 (24%) articles. Only seven (7%) articles reported on multi-country studies. Figure 3 provides a more detailed description of the focus countries.
Research focus and study population
We categorized articles in relation to their research focus according to the standard HPSR classification (Gilson 2012). 50% of all retrieved articles (52 out of 105) addressed topics related to Health Systems, with Service Delivery (16), Human Resources (13) and Financing (12) accounting for most publications. Health Policy accounted for 42 (40%) articles and Health Outcomes for 11 (10%) articles. Among the Health Policy articles, the vast majority (37 out of 42) addressed either Implementation or Evaluation (see Figure 4 for details).
The research focus on mixed methods HPSR did not vary over time, with the proportion of articles being classified under each theme remaining stable over time. Although research focused on Health Systems was present across all regions, research on Health Policy appeared only in studies conducted in East, West and Southern Africa. During the last observation period (2010–13), in East and West Africa, studies focused on Health Policy prevailed over studies focused on Health Systems.
Three-fourths of all articles (78) reported on studies targeting one single set of respondents (health workers, communities, patients/clients, policy makers), while the remaining one fourth reported on studies addressing multiple constituencies of respondents. The proportion of studies including multiple respondents did not increase over time.
Clarity of research question and study design
Although nearly all studies (96) were explicitly defined as mixed methods studies in the text of the manuscript, only slightly above half of all of them (55) reported a clearly stated mixed methods research question. As an illustration for clearly stated mixed methods research question, we wish to refer to a study looking at reasons for bypassing primary healthcare facilities for child care in rural Tanzania (Kahabuka et al. 2011). Already at the end of the background section, the authors clearly state the study objectives in relation to: (1) ‘establish the frequency of bypassing such facilities’; (2) ‘identify factors influencing bypass’; and (3) ‘explore experiences, at such facilities, of caretakers seeking care for their under-five children’. By combining ‘establish the frequency’ and ‘identify factors’ as juxtaposed to ‘explore’, the authors set the tone for a mixed methods study from the very onset of the article, allowing the reader to already differentiate quantitative from qualitative aspects of inquiry.
Similarly, only 18 studies (17%) referred to a specific mixed methods design. Only two out of these 18 studies were published before 2010, suggesting that the practice to define a study in relation to a specific mixed methods design has increased over time. References to study design were mostly aligned with the classification proposed by Creswell and Plano Clark (2011), although in seven out of these 18 articles, timing of data collection (concurrent vs sequential) rather than the function and role of mixing methods (triangulation, explanatory, exploratory) was used as the exclusive criterion to define study design. Similarly, only six out of 105 articles provided any detail on the relative weight of the quantitative vis à vis the qualitative study component, that is to say on the relative importance/focus attributed to one component as compared to the other.
Quantitative study components
Over 90% of all retrieved articles (97) relied on a single quantitative data collection method, most frequently a structured close-ended survey (81). Sixteen articles (15%) relied on secondary data, while only two articles reported on a randomized controlled trial. We did not detect any consistent differences in data collection methods across respondent types.
Out of the 105 retrieved studies, 84 (80%) clearly described the quantitative sampling strategy. The sampling strategy was either not described clearly (and therefore could not be appraised) or was not adequate vis à vis the research question at the core of the study in 21 out of 105 reviewed articles. Out of the 84 studies where quantitative sampling techniques were described clearly, nearly one third relied on mixed methods sampling rather than pure probability sampling. As an illustration, let us look at a study conducted to assess and explore barriers to access to care for women in South Africa, where the authors purposely selected the sites where to carry out their survey (two urban and two rural sub-districts) and then selected 300 women at each site, based on an expected size of socio-inequalities in utilization of delivery services.
Only half of the 105 articles reviewed of all articles included a clear description of the quantitative analytical approach. Similarly, only in 62 out of 105 studies, the analytical strategy adopted provided sufficient evidence to answer the research question being addressed. Limitations to analysis often arose due to relatively small samples, which made it impossible to run complex statistical models. In turn, small samples were related to the mixed methods nature of the study, especially in cases when quantitative and qualitative data collection strategies were integrated, as it was the case in a study reporting on the performance of community health workers in Uganda (Kalyango et al. 2012).
Qualitative study component
More than half of all reviewed studies (54) included at least two qualitative methods of data collection and analysis, with 17 out of 105 studies including at least three methods. The most frequently used methods of qualitative data collection were Focus Group Discussions (FGD) and In-depth Interviews, each used in nearly two-thirds of all qualitative study components. Other methods of data collection, specifically observation and document review, were reported in < 10 studies. Qualitative data collection was integrated with quantitative one in four instances, when open-ended questions were integrated in a structured quantitative survey. In three out of these four cases, this represented the exclusive means of collecting qualitative data.
In 68 (64%) out of 105 studies, qualitative sampling strategies were described in sufficient detail, with the strategy being aligned with the research question and thus defined as adequate in 65 out of these 68 studies. The vast majority of studies with a clearly described sampling strategy (56) applied non-probability sampling, with purposive sampling representing the most frequent option (50) and convenience sampling being selected in only six cases. An illustration of a careful purposive sampling approach comes from the work of Spangler and Bloom, who engaged in several months of field observation before selecting women with a recent history of childbirth to be included in the qualitative component of their study in Tanzania (Spangler and Bloom 2010). Four studies relied on probability sampling for their qualitative component, but these four studies are not the same four studies (Yassin et al. 2003; Mullei et al. 2010; Shattuck et al. 2011; Bhopal et al. 2013) that relied on open-ended questions in structured survey for qualitative data collection.
The qualitative analytical strategy was clearly described in 69 (65%) out of 105 studies. Thematic analysis was by the far the most frequently used qualitative analytical approach used in 34 studies and followed by content analysis used in 15 studies. Grounded theory, standard comparison method, and the framework approach were each used in a handful of studies.
Integration across quantitative and qualitative findings
In 65 (62%) out of 105 studies, quantitative and qualitative findings were presented separately, although in nearly all studies (101), they were discussed in an integrated manner. In the vast majority of cases (82), the level of integration in the discussion of the finding was coherent with the research question. As an illustration, let us refer again to the study assessing and exploring barriers to access to care among women in South Africa. Both the methods and the results section are organized by study component (QUAN and QUAL), while the discussion clearly integrated the two and draws inferences based on a joint interpretation of quantitative and qualitative findings (Silal et al. 2012). The study by Spangler and Bloom (2010) takes it even a step forward, carefully integrating study components already at the level of the description in the results section, probably due to the flexibility allowed by the journal (Social Science and Medicine) where their work is published. Still, only about one out of every four studies (23) discussed the limitations of this process of integration and/or any apparent divergence between quantitative and qualitative findings. Similarly, only one out of every ten studies (10) addressed concerns pertinent to the researcher’s bias in conducting the research and interpreting its results. Half of all studies (52) discussed findings in relation to context. Overall, two-thirds of all studies (70) appeared to adopt a design and collect sufficient data adequate to answer the initial research question.
Table 1 reports on the summary characteristics of mixed methods quality assessment we carried out as part of our review.
Discussion
This review represents the first step towards systematically appraising the application of mixed methods in HPSR in Africa. More in general, this review represents one of the very first attempts to appraise the quantity and nature of mixed methods research in global health, moving away from existing reviews focused on synthetizing evidence on single health issues and from methods-focused reviews drawing exclusively from work conducted in high income settings (O’Cathain et al. 2008; Östlund et al. 2011). As indicated earlier in this manuscript, we purposely abided to a broad definition of mixed methods research, including gray area articles. In line with the arguments already raised in the introduction, its intrinsically interdisciplinary nature (Creswell and Plano Clark 2011), and its adoption of pragmatism as the leading research paradigm (Johnson and Onwuegbuzie 2004) make mixed methods research ideal for application in the field of HPSR (Robert and Ridde 2016).
The immediate most striking finding relates to the increase in the number of publications that is observed over time, with over half of all reviewed studies being published after 2010. This clearly indicates how mixed methods research is gaining momentum, acquiring more prominence and value in HPSR (Ridde and Olivier de Sardan 2015). The increase we observed is probably a reflection of a shift in research culture, moving from a single-method tradition towards a multi-methods one, at least within the field of HPSR. In turn, this shift in research culture could have been motivated by an increasing need to address multidimensional research questions, often reflecting the complexity of the systems and the interventions being studied. Additionally, this shift in research culture is likely to have been facilitated by a number of publications, such as the HPSR Methodological Reader (Gilson 2012), as well as by the recent calls in favour of the application of mixed methods put forward by leading research institutions, including the Rockefeller Foundation (Bamberger 2012), the NIH (Creswell et al. 2011) and the MRC (Moore et al. 2015). It is still unclear to what extent the growing call for mixed methods is matched by increasing funding opportunities for this type of studies. The specific focus of the articles reviewed as well as the study location do not appear surprising, as they largely reflect the overall academic production within the field of HPSR (World Health Organization 2012). It is possible that had we included disease-specific studies, we would have ended up detecting a larger number of studies focused on health outcomes. Our strict focus on HPSR meant that we only identified a handful of studies focused on health outcomes, defined primarily in relation to maternal and child (especially nutrition) outcomes.
Our review, however, also highlights that only a small proportion of the reviewed studies abided to the quality standards set for mixed methods research (Pluye et al. 2009; Creswell and Plano Clark 2011; Creswell et al. 2011; Moore et al. 2015). These findings are aligned with prior evidence, suggesting lack of conceptual clarity and important methodological shortcomings in HPSR (Mills 2012). For instance, just above half of all reviewed studied described as starting point a clearly defined mixed-methods research question, that is to say a question that allows the researcher, and afterwards the reader, to identify and distinguish the quantitative and qualitative elements of inquiry (Creswell and Plano Clark 2011). Similarly, only one every five studies clearly defined a mixed methods design (Creswell and Plano Clark 2011), explicitly stating the purpose for integrating quantitative and qualitative methods of data collection and analysis within a single research effort, although in the majority of reviewed publications, quantitative and qualitative findings were later integrated in a coherent discussion.
This observation suggests that researchers in the field of HPSR may increasingly turn to mixed methods research as a way of responding to an increasing need to address complex research questions and to a parallel collective call to do so using mixed methods approaches, although they may not yet be fully conversant with their conceptual underpinning. In turn, this opens the door to the risk that mixed methods research is not conducted according to its highest quality standards and that researchers may therefore fail to seize the full potential for advancing knowledge generation beyond the boundaries of single-method approaches. Mixed methods research should not be pursued as a fashion at the expense of quality, but should rather be pursued as a distinct and sound form of inquiry when the research question obviously calls for it (Tashakkori and Teddlie 2010), engaging researchers to be self-reflective about their own work.
In their work, both Creswell and Plano Clark (2011) and Greene (2007) recognize that most researchers lack specific expertise in mixed methods research. Most academics approaching the field of HPSR are originally trained in one single discipline, ranging from epidemiology to economics, from medicine to anthropology. As such, they are normally exposed to a discipline-specific methodological modus operandi, which in turn is based on a discipline-specific theoretical paradigm, as discussed in a seminal paper by Coast et al. (2004) already in 2004. The mere will to move from a single-method to a multi-method approach to data collection and analysis does not automatically translate into the ability to effectively do so. This ability to move from a single-method to a multi-method approach relies on the capacity both to bridge across theoretical paradigms (Tashakkori and Teddlie 2010), accepting that multiple mental models are possible (Greene 2007), and to acquire sufficient expertise in the application of a given method in actual research practice (Creswell and Plano Clark 2011).
Unfortunately, the offer in mixed-methods training in the Health Sciences is still very limited, especially when considering a specific focus on LMICs, including Africa. A recent Global Online Mapping of HPSR training (Tancred et al. 2015) in 169 different organizations from 59 countries failed to provide any data about mixed methods research training within the framework of HPSR courses taught in LMICs, possibly suggesting that such a focused mapping would be much needed. We are aware of courses in mixed methods being taught in Canada, both at McGill and at Toronto University, but these courses do not focus on LMICs. The only course in mixed methods in the Health Sciences with a specific focus on LMICs was identified at Heidelberg University (Germany), but the course engages a limited number of highly selected students and is therefore not sufficient to respond to increasing demands in the field. A similar course is taught every year at the Institute of Tropical Medicine in Antwerp (Belgium), but with no specific focus on LMICs. Moreover, participation to these courses is subject to the payment of high fees, which represent an additional barrier to access adequate training, especially for researchers residing in LMICs. Still, this lack of training in mixed methods research in the Health Sciences is not surprising given that capacity to carry out mixed methods research is not valued as a core competency for master students in public health or global health by the Association of Schools of Public Health (ASPH) (ASPPH 2017; Core Competencies in Public Health 2017). In turn, this is suggestive of an obvious mismatch between the call for the wider application of mixed methods research advanced by the HPSR community (and supported by funding agencies) and formal academic curricula in public and global health. It follows that there is an obvious need to re-align the two, possibly also through the development of online courses in addition to more traditional face-to-face training products.
Beyond the lack of specific training in mixed methods research, we postulate that some of the weaknesses we have come across in the studies included in our review, such as lack of clarity on sampling, data collection and analytical strategies, may not be due to actual weaknesses in the conduct of the studies, but rather to a combination of journal structures imposing strict word limits and lack of clear guidelines on how to report mixed methods evidence. Similarly, reviewers and editors, often people with specific expertise in either quantitative or qualitative methods, may lack access to guidelines on how to handle and assess mixed methods publications. We have often been faced with the challenge of reporting evidence from their mixed methods studies in a single manuscript, when leading journals cap an article length at 3000–4000 words and impose a strict classical structure (introduction, methods, results and discussion), which is not always compatible with the emerging nature of mixed methods designs (Creswell and Plano Clark 2011). We are aware of two journals with an explicit focus on mixed methods research, allowing also for longer manuscripts: the Journal of Mixed Methods Research and International Journal of Multiple Research Approaches. Both these journals, however, do not address the health sciences specifically and are almost exclusively focused on methodological reflections in mixed methods research. Thus, researchers in HPSR in Africa have little stake in using these journals to disseminate their work. Calls have recently been made to relax the structural requirements imposed by journals to allow for the publication of more qualitative research (British Medical Journal Publishing Group 2016; Social science approaches for research and engagement in health policy & systems (SHaPeS) thematic working group of Health Systems Global, Regional Network for Equity in Health in East and Southern Africa (EQUINET), and Emerging Voices for Global Health et al. 2016), and we advance that this concern is equally, if not more relevant, for mixed methods research. Although not sufficient to overcome the barrier imposed by the journal structural requirements, a number of authors have recently worked to develop guidelines to promote standardized approaches in the dissemination of mixed methods studies (Leech and Onwuegbuzie 2010; Wisdom et al. 2012). We trust that these guidelines will be useful to ensure that, unlike what observed in our review, future studies will report all needed details on design, sampling, and analytical approach.
Methodological considerations
Albeit very valuable as the first review focused on assessing the overall scope of mixed methods research in the field of HPSR in Africa, our work suffers from a number of limitations. In the first place, for pragmatic reasons, we only included peer-reviewed articles published in English, while it is possible that mixed methods studies could have also been released in French, Arabic, or Portuguese, e.g. having limited the review to articles published in English is likely to have implicitly influenced the distribution of studies we recorded across SSA regions, with English-speaking countries, such as South Africa, Uganda and Ethiopia counting higher numbers of publications than French, Arabic, or Portuguese speaking countries. It is possible in fact, that research teams working in these settings are more likely to publish in French or in Portuguese than in English, due to a wish to reach out to their own policy community, but possibly also due to lack of access to English-speaking training opportunities. Second, to limit the scope of our review to a manageable size, we purposely included only studies that addressed overall policy and health systems questions and excluded studies based on single diseases, such as malaria. This obviously means that our review is only reflective of strictly-speaking HPSR literature, but that more mixed methods research is likely to be ongoing in more disease-specific research fields. Third, and probably most importantly so, we need to acknowledge that the real scope of mixed methods research in HPSR is likely to be much wider than what could be captured by our review. It is very likely in fact that research teams adopting a mixed methods approach in their work, end up writing reports (and not necessarily peer-reviewed manuscripts) or end up publishing quantitative and qualitative results separately, in order to comply with journal regulations. This obviously entails a loss of fidelity in the application of mixed methods, since the interpretation can never be as rich if quantitative and qualitative sets of data are kept separate. Similarly, we cannot exclude having missed to identify some mixed methods publications, if not clearly identified as mixed methods or as combining quantitative and qualitative findings already in the abstract (such as for example, realist evaluations).
Conclusions
This review has shown that the number of mixed methods studies with a focus on HPSR in Africa has increased substantially over the last few years, but the important gaps in the quality of the studies and their reporting need to be filled. The review points at the importance of providing adequate training in mixed methods and calls for direct investments in this direction. Along these lines, it would be desirable that similar reviews are conducted on the state of mixed methods research in other continents and in relation to other areas of global health research, to draw a more comprehensive picture of mixed methods experiences globally. Lastly, the review points at the difficulty faced by researchers wishing to disseminate their mixed methods work through existing journal formats. This raises a question as to whether leading health journals should not relax their requirements (especially in terms of length) to accommodate other types of studies and possibly calls for the creation of a new journal devoted exclusively to the publication of mixed methods research in the Health Sciences from researchers working in LMICs.
Supplementary Data
Supplementary data are available at Health Policy and Planning Online.
Acknowledgements
The authors are grateful for the support of the study librarian, Daniela Ziegler, who assisted in identifying and retrieving the needed literature. Is received a training bursary from the Fonds de recherche du Québec-Santé (FRQS). This research project is part of the ‘Community research studies and interventions for health equity in Burkina Faso’ funded by the Canadian Institutes of Health Research (ROH-115213). V.R. holds a CIHR-funded Research Chair in Applied Public Health (CPP-137901). M.D.A. is employed as permanent Faculty at Heidelberg University and Gilbert Abiiro completed this work while working in her team. Sponsors did not have a role in the study design; the collection, analysis and interpretation of data; the writing of the report; and the decision to submit the article for publication.
Funding
Manuela De Allegri is permanent faculty at University of Heidelberg. Her contribution and that of Gilbert Abiiro were supported directly by University funding. The contributions by Valery Ridde and by Isidore Sielenou were supported by the Canadian Institutes of Health Research, which funded the program (ROH-115213) “Community research studies and interventions for health equity in Burkina Faso”.
Conflict of interest statement. None declared.
References
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, et al. (2009). The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. Plos Medicine 6(7) : e1000100. doi:10.1371/journal.pmed.1000100