Producing Change to Understand the Social Determinants of Health: The Promise of Experiments for Social Epidemiology

Abstract

In this commentary, invited for the 100th anniversary of the Journal, we discuss the addition of randomized experiments, along with natural experiments that emulate randomized trials using observational data, as designs in the social epidemiologist’s toolbox. These approaches transform the way we define and ask questions about social exposures. They compel us to ask questions about how well-defined interventions change a social exposure that might lead to changes in health. As such, experiments are of unique public health and policy significance. We argue that they are a powerful approach to advance our understanding of how well-defined changes in social exposures impact health, and how credible social policy reforms may be instrumental to address health inequalities. We focus on two research designs. The first is a “pure” randomized controlled trial (RCT) in which the investigator defines and randomly assigns the intervention. The second is a natural experiment, which exploits the fact that policies or interventions in the real world often involve an element of random assignment, emulating an RCT. To give the reader our bottom line: While acknowledging their limits, we continue to be very excited about the promise of RCTs and natural experiments to advance social epidemiology.

social epidemiology, social experiments, social policy

Abbreviations

 
• ENRICHD

  Enhancing Recovery in Coronary Heart Disease

 
• MI

  myocardial infarction

 
• RCT

  randomized controlled trial

Randomized experiments, along with natural experiments that emulate randomized trials using observational data, are increasingly common designs in social epidemiology, which have transformed the way in which we define and ask questions about social exposures. They compel us to ask questions about how measurable and actionable changes in social exposures (e.g., an additional year of schooling or a $500 monthly cash transfer) affect health. Observational studies in social epidemiology traditionally focus on identifying social “risks factors,” but they rarely aim to identify the causal impact of concrete social policies or interventions. Thus, experiments are of unique public health and policy significance: They offer a powerful approach to advance our understanding of how well-defined social and structural changes affect health and how social policy reforms may be instrumental to address health inequalities. Our main arguments apply to experiments and quasi-experiments beyond social policies (e.g., natural disasters, political events); however, we focus on social policies because of their relevance for a social epidemiology that can inform action to address health disparities.

Prior literature has articulated excellent conceptual and methodological insights on randomized controlled trials (RCTs) and natural experiments in social epidemiology. For example, Diez Roux (1) argues for a practical approach that conceptualizes social causes in a continuum from “upstream” distal factors to specific interventions on narrowly defined factors, where only the latter maybe studied through experiments. Glymour et al. (2, 3) offer excellent theoretical and technical resources for understanding key assumptions and technical issues that arise in natural experiments and especially in instrumental variable analysis. In this commentary, we aim to do something different by reflecting on what we have learned in the practice of doing RCTs and using natural experiments to address social epidemiology questions. In particular, we aim to draw attention to challenges that arise in the design and interpretation of an experiment to test the impact of a social determinant on health and in the exciting yet grueling task of evaluating the impact of policies using natural experiments. As we shall argue, drawing conclusions from experiments on the causal impact of social determinants on health is not straightforward, and requires us to think very carefully about thespecific nature of the intervention tested in an RCT or specificities of social policies evaluated in natural experiments. More often than not, we have identified unintended consequences of interventions tested with an experimental design, which result from not thinking through some pitfalls in design or analysis. This commentary focuses on our experiences in the field and some lessons learned from counterintuitive results.

In this essay, we focus on two research designs. The first is a “pure” RCT, in which the investigator defines and randomly assigns the intervention. The second is a natural experiment, which exploits the fact that policies or interventions in the real world often involve an arbitrary cutoff assigning people to “treated” and “control” groups, thereby emulating an RCT. To give the reader our bottom line: We continue to be very excited about the promise of RCTs and natural experiments to advance social epidemiology. At the same time, we acknowledge the limits of experimental work to address certain types of questions that are best suited to observational studies. Important issues we will cover in this piece relate to: 1) the length of exposure needed to measure health impacts, as well as the identification of appropriate etiologic periods; and 2) the tension between evaluating the impact of a narrow or specific intervention vs. identifying impacts of long-run social exposures on health. The effect of exposures such as structural racism or poverty accumulate over long periods and affect multiple dimensions of well-being (e.g., employment, health). Experiments are better suited to assess “acute” interventions (e.g., income transfers) or specific policy changes (e.g., abolition of Jim Crow laws). It is the scope of what they assess that differentiates them, and makes the approaches complementary.

RANDOMIZED CONTROLLED TRIALS IN SOCIAL EPIDEMIOLOGY

After 10–15 years as a social epidemiologist working with observational data from community-based cohort and national studies, I (L.B.) decided to make an important turn toward experimental approaches in the mid-90s as a way to understand the impact of social determinants on population health. I did this for two reasons. First, the experimental design rests on a powerful framework for identifying causes of poor health, and second, but perhaps more honestly and importantly for me, it shows us how to intervene to improve population health. Public health rests on both acts of identifying causal exposures and taking action to address those causes to improve population health and reduce health inequities (4). Experiments enable us to learn how generating change in social exposures can improve health. They also compel us to identify a theoretically plausible sensitive or critical period in the life course and test the hypothesis that intervening on that social exposure in that period would lead to changes in health. In many ways, this course correction has been incredibly rewarding, while in other ways, it turned out to be naive. The shortcomings of RCTs recently highlighted by economists, sociologists, and epidemiologists are important to note (5–7). We focus here on challenges specifically in the context of social epidemiology.

My first foray in experiments was Enhancing Recovery in Coronary Heart Disease (ENRICHD), a multicenter randomized, double-blind clinical trial to reduce depression and improve social support to reduce myocardial infarctions (MI) and mortality in men and women who had already had their first MI (8). It was designed as a psychosocial intervention. The study design was strong, methods impeccable, and the study population was diverse in terms of race, ethnicity, and gender. The intervention was based on individual cognitive behavioral therapy oriented to reducing social isolation and/or depression. Each participant received weekly then monthly visits by a professional therapist over a 6-month period. What did we learn from this major psychosocial experiment? On the positive side, we learned we could recruit and maintain a large diverse sample of men and women (n = 2,481) and that in fact we could modify levels of depression and social support within 3–6 months. However, intention-to-treat (ITT) estimates revealed a null impact of the intervention on major outcomes of interest, although post hoc analyses showed some effects for White men. The fact that we observed a null effect forced us to spend time thinking about why a very comprehensive intervention in a well- designed trial did not lead to expected outcomes.

In this process, we learned several lessons. First, we were very unclear about the etiological period and the point in the life course at which the intervention could change levels of depressive symptoms or social support, which in turn would lead to changes in cardiovascular disease. Changing these conditions after someone had their first MI may well have been too late. Experiments gain part of their power not only from assessing causal relationships but also from identifying the etiologic period at which modifying risk will—or will not—modify the outcome. This issue was evidenced in the case of reinfarction and death, as a participants’s risk for these outcomes has a small window of high risk after MI. In our trial, the mortality rates following MI were undergoing a rapid transition, dropping in risk virtually every year. Our intervention took months to conduct and was completed well after this period of highest post-MI mortality risk. In terms of identifying events, it seems as if we missed the most important period of risk. With hindsight, it may seem obvious. Yet, it is surprising how these issues continue to plague RCT studies on the impact of social interventions.

A second lesson refers to the importance of understanding the context of a specific RCT, including the barriers participants face. In ENRICHD, our intervention did not contextualize experiences of isolation for those in disadvantaged positions or who faced structural barriers, including structural racism. The model of ENRICHD was an individualized and classically biomedical intervention, rather than a social intervention to reframe the social environment. While we acknowledged that social isolation and lack of support was a social experience, we did not orient our intervention to the social environment. We learned that the level of intervention (whether at the individual, group, or population level) is a critical decision. For example, the fact that we observed some effects only on White men in ENRICHD, but not in Black or Hispanic participants (9), may reflect structural barriers that the latter groups face and that limit the success of the intervention to improve health outcomes.

Finally, there is of course the very real possibility that the trial was a more rigorous test of causal relations than an observational study, and neither depression or social isolation is causally related to cardiovascular disease. Given that there are now multiple meta-analyses showing that both social isolation and depression are related to cardiovascular disease and mortality (10, 11), this raises the question of why there are differences between the ENRICHD RCT and observational studies. The substantive limitations outlined above likely play a role, as do analytical differences between RCTs and observational studies (12). However, this also reiterates our previous point: The impact of a short-term and very specific intervention well-suited to evaluation using an RCT design may not offer a full answer to the question of how cumulative exposures that result from long-run social isolation and depression have an impact on health.

My second large foray into social experiments was the Work, Family and Health Network (WFHN). Lessons learned from ENRICHD about the importance of the level of intervention suggested that we should focus on organizational-level change rather than individual change. The WFHN trial aimed to change the social environment rather than the individual. It did this by intervening with multiple team members through a supervisor-level intervention and simultaneously a group-level, organizational-change team of employees that gave workers a voice in workplace redesign (13). The organization was identified as the right intervention level through other worksite interventions launched at the time of the WFHN trial (14, 15), as well as pilot data showing that social circumstances could be modified at that level (16). Results suggest that the intervention improved sleep, reduced tobacco consumption, and had an impact on short-term cardiovascular risks (17–19). Intervening on the social environment got us closer to the type of social intervention that is likely to lead to a meaningful health change. In subsequent analyses, we also learned that the effect of an intervention can differ dramatically across different industries that operate in vastly different contexts. In particular, the impact of the intervention was often stronger for the information technology industry, which was more flexible and better able to implement organizational change for work-life balance and schedule control. The long-term care industry, on the other hand, was highly regulated and had less organizational flexibility, affecting our ability to produce change.

In addition to trials we have designed, we have learned a great deal from existing social experiments in the areas of education, income, employment, social assistance, or housing. The United States has a rich history of testing new social policies using an experimental design (20). This is an untapped resource for social epidemiologists, and recent research has built on these experiments to incorporate health outcomes into their evaluations or to link them to routinely collected health and mortality data (21, 22). One example is the Opportunity NYC–Family Rewards RCT (2007–2010), which offered cash transfers to low-income families in New York City’s most deprived communities conditional on their participation in education, preventive health care, and parental employment. The intervention led to a large increase in the use of preventive dental care and modest increases in health insurance coverage and parental self-reported health and levels of “hope” among parents. However, it had no or modest effects on the health of parents and children (23).

A first lesson from Opportunity NYC–Family Rewards is that RCTs do not operate in a vacuum: The NYC intervention operated alongside a range of long-standing social programs, including the Earned Income Tax Credit, Medicaid, and substantial housing support. The counterfactual, therefore, was not the absence of financial support but a rather comprehensive set of social safety net programs, which were minimally enhanced by the addition of cash transfers in the treatment group. This may explain why we observed relatively weaker effects of this conditional cash transfer program in NYC relative to Latin American countries, where these transfers showed stronger benefits but where the counterfactual was no or limited social benefits. Second, a key feature of Opportunity NYC–Family Rewards was a parental employment incentive. As the Great Recession happened in the middle of the intervention, many adult participants were not able to gain or maintain full-time employment, a key outcome on which long-term health and well-being effects had been predicted. A third lesson is that experiments draw their conclusion from a narrow definition of change caused by a very specific social intervention. We estimated the effect of a conditional cash transfer received for a period of 3 years, which is not the same as an intervention that would secure stable income from benefits or employment for a longer period.

To summarize, what we learned from these trials is that they enable us to address critical questions in social epidemiology by pointing to the impact of specific social policy changes that affect the social determinants of health. They also highlight the importance of moving from intervening at the individual level to intervening at the social environmental level at which we theoretically think change can occur. This movement more deeply acknowledges that to produce structural change we need to intervene at this “structural” level. This is a critical lesson that is now more commonly understood and implemented. Second, we learned that timing is critical both in terms of the etiological period and, potentially, the length of the intervention. Decisions on timing are made in the absence of complete information but are increasingly informed by pilot studies and early studies in the area. Third, heterogeneity in treatment effects point to the importance of the larger social context in which interventions are implemented, which either enhances or limits the opportunities for success of the intervention. Here we suspect we will always find important and informative variation in effects that will guide future interventions. Fourth, we highlight the importance of understanding the counterfactual—represented by the control group—in interpreting the findings of RCTs. These lessons all point to careful planning and theory-driven thinking that can enhance the success of trials.

NATURAL EXPERIMENTS TO ADDRESS SOCIAL EPIDEMIOLOGY AND POLICY QUESTIONS

Natural experiments share with RCTs the fact that the mechanism of treatment assignment is random or quasi-random, but unlike RCTs, researchers have no control on the intervention and they rely on ingenious ways of analyzing observational data. The ideal natural experiment is one in which governments or other actors randomly—or quasi-randomly—assign individuals or groups to receive a program or policy. Examples include lotteries used by the US Government for military conscription for the Vietnam war (24) or the state lottery used in Oregon in 2008 to assign low-income residents to get Medicaid insurance (25). Natural experiments with full randomization are rare, however, as policy makers most often target policies based on need. This poses a critical challenge to scientists interested in the impact of policies: As the mechanism of assignment is based on need, treatment and control differ in critical social and economic outcomes to start with. To assess policy impacts, therefore, we cannot simply compare health outcomes of beneficiaries and nonbeneficiaries, as these two groups are, by design, not exchangeable.

Natural experiments offer a solution by encouraging scientists to look closely at the policy and identify groups that differ only by virtue of their eligibility for the policy. They are exciting because they enable us to evaluate the impact of policies that have already been scaled-up or implemented but for which we are not fully aware of the health impacts. They can provide “real-world” evidence of effectiveness in ways RCTs cannot. On the other hand, they are challenging because they require a detailed understanding of the policy design and implementation, and they rely on our ability to identify a mechanism of treatment assignment deemed as good as random. Some mechanism of random or quasi-random assignment can nevertheless often be found even for the most elusive policies, as they often affect some, but not all members of a population, and eligibility is often based on arbitrary factors unrelated to the outcome of interest. For example, a policy reform may affect only cohorts born after a presumably random date. The key assumption we make here is that those who are eligible are not different from those not eligible, apart from being subject to the reform. Let us take the example of compulsory schooling laws to illustrate this point and reflect on key lessons we learned from over a decade of using these approaches to answer social epidemiology questions. Compulsory schooling laws have been widely used in economics and social epidemiology to study causal effects of law-mandated changes in schooling duration on a range of social and health outcomes (26, 27). They “tick all the boxes” of “good” natural experiments: well-defined policy measures, with a clear threshold for eligibility based on a variable (date of birth) that is well measured in surveys. We and others have exploited the fact that we do not expect people born just before cutoffs for eligibility to longer schooling to be different from those born just after, apart from exposure to the policy. What have we learned from these studies?

The first lesson is that results from natural experiments can differ from results from observational studies using “conventional” regression methods. Decades of observational research has shown that more years of schooling and higher levels of education are associated with better mental and physical health in adulthood. However, a concern is that it is not education but rather other factors associated with educational attainment (e.g., family background, cognitive ability, or genetics) that explain why people with higher education have better health. In our work, we tried to address this issue by analyzing long-term effects on health of a compulsory schooling reform implemented in France (28, 29). Implemented in the 1950s, this reform increased duration of compulsory schooling by 2 years for cohorts born after January 1953 (treatment group) but left unchanged the years of compulsory schooling for cohorts born before that date (control group). We found that the reform increased years of schooling as intended but its health effects were mixed. It improved cognitive function among men, but it increased depressive symptoms among women (28) and led to poorer biological profiles in respondents who were born in low-income families (28). These findings are puzzling: Shall we conclude that more years of schooling can lead to poorer health?

An alternative explanation brings us to the second lesson: As we are using historical policy changes, we do not get to design policy reforms and are left with what was decided at the time, however imperfect or difficult to measure that may be. In the case of the French reform, further research on the policy showed that it increased time spent in class but not much more as the reform was not supplemented with broader infrastructural investments or curriculum reform. It did not increase likelihood of getting a higher degree and did not translate into higher earnings or rates of employment in adulthood, contrary to reforms in other countries. Increasing the quantity of education alone may not have been effective, and it may have been counterproductive for some groups (30). In work using a similar reform in the United Kingdom, we found that young people who were forced to stay at school longer but wished they could have left early had worse mental health across the life course than young people who were also required to stay longer but wanted to stay at school (31). The reform negatively affected individuals who felt they lost a year of work experience or vocational training. Of course, these policies may have different health effects for historical cohorts than they would if they were implemented today in the same country. For example, in the French reform, women eligible for longer schooling faced considerable challenges to converting additional years of schooling into better diplomas and jobs in the late 20th Century. However, women benefitting from the same policy in the 21st Century may be better able to reap the benefits of longer schooling, given women’s advancement in the labor market during the last decades. Again, context and access to other resources matters. The literature on compulsory schooling reforms illustrate the trade-offs between increasing our ability to make causal claims by studying a well-defined policy intervention and losing sight of the bigger picture by focusing on a very specific (and limited) way to intervene on education.

A third lesson is the importance of interdisciplinary collaborations with the social sciences, economics, social policy, and law. This is also true for the investigation of the health effects of social experiments, as we learned in the NYC Family Rewards program discussed above. Interdisciplinary collaborations are essential to understand the theory, mechanisms, and implementation of historical social policies. For example, collaboration with economists was essential in order to understand the complex social and economic impacts of compulsory schooling laws (28, 29, 31), the earned income tax credit (32), and maternity leave benefits (33) and their consequences for health. Interdisciplinarity is critical because natural experiments require a full understanding of social policy reform in order to identify a control group that offers a good counterfactual. Identifying a good control group requires expert knowledge of the laws underlying a policy change and historical research on how the policy was implemented. Another dimension of these necessary collaborations is that the implementation of quasi-experimental methods requires a good understanding of causal inference theory and econometric approaches. Training is a key challenge that we have encountered in our own journey trying to analyze these policy reforms and their consequences for health, especially as the field evolves rapidly as econometric methods become more and more sophisticated. Our own training was often ad hoc, and we were lucky to work in interdisciplinary departments. We hope that approaches such as regression discontinuity designs, instrumental variables, and difference-in-differences (and importantly their assumptions and limitations) will become a mainstay of social epidemiology training.

Finally, although natural experiments rely on the strength of the research design, a fourth lesson we have learned is that a strong design is not enough; logistics are also key. For example, a natural experiment requires that we have data at the right time and for the right populations. Often, this means data before and after a policy was enacted and implemented, or information that enables us to distinguish individuals affected by the reform from seemingly comparable individuals who were not affected by the reform.

Bearing in mind these challenges, we cannot but emphasize the huge value of natural experiments. They offer a rigorous yet pragmatic way to understand how policies make an impact on the health of populations. Under certain assumptions, they enable us to study the causal impact of policies in “real life,” unlike RCTs, which often require us to deliver policies in limited ways that differ from real-world practices. They also enable us to evaluate the impact of policies that we think may have negative effects on health and inequalities but that cannot be randomized for ethical, political, or practical reasons, such as mass incarceration or welfare benefit cuts. Importantly, even if only partially, they also enable us to study the impact of deep societal changes, including those arising from major social transformations such as the French revolution (34), the fall of the Berlin wall (35), or exposure to armed conflict (36).

CONCLUSIONS: THE STRENGTHS AND LIMITATIONS OF EXPERIMENTS FOR SOCIAL EPIDEMIOLOGY

We reflect on the value of experiments in social epidemiology as the Journal celebrates its 100th anniversary. Looking back at our own work and the work of others, it is clear to us that social experiments are a powerful research design to test novel hypotheses on the impact of changing social determinants of health through interventions and policies. They enable us to evaluate health impacts of many social and economic policies that were not explicitly designed to improve health. Those spillover effects on health have often been unrecognized and therefore not “counted” as real benefits (or costs). Yet experiments need to be grounded in social theory and build on observational research that points to theoretically plausible social exposures, interventions, mechanisms, and etiological periods. The interplay between theory, observational research, and experiments is therefore critical to advance social epidemiology.

There is a danger in going too far in making causal claims based on experiments that test interventions that have weak or modest effects on the very social determinants whose health effects we are interested in understanding, such as poverty or education. In these situations, we are unlikely to get a full picture of the complex causal mechanisms by which these social determinants affect health. Using experiments to advance knowledge on how social and economic factors influence health would require us to have more powerful interventions that can effectively change social and economic determinants. This is not a limitation of experiments per se. Instead, it is a limitation of our knowledge base on effective social policies and interventions that generate the transformative changes we need to produce to understand the “full” impact of social and economic exposures on health. Often, we overreach here as we extend “meaning” from a very specific intervention to a larger social condition.

This leads us to a critical distinction between experimental and observational research in social epidemiology: Examining the health impact of a change in a social determinant (e.g., income, education) produced by a social policy or intervention is not the same as examining the impact of that social determinant on health. Social policies and interventions produce, in most cases, only a modest change in social determinants. This raises the following question: In which cases are experiments useful to study the health impact of social determinants? Conceptualizing social causes as a continuum, Diez Roux (1) argues that experiments are appropriate to study the impact of a narrowly defined “proximal” factor on which we can intervene but less useful to assess the impact of “upstream” distal factors. Our argument is somewhat different: We argue that social causes might be better distinguished by the breadth of social experience and time of exposure rather than proximal vs. distal. Back to our example on compulsory schooling laws, we are approaching the effect of education through a 1- or 2-year increase in the duration of schooling in adolescence, which cannot reveal the impact of a more transformative change that would profoundly modify complex educational trajectories and experiences. This is even more critical when we try to study social exposures like structural racism using these tools. Changing Jim Crow laws, while focusing on a structural and “distal” determinant (the law), is a narrower and more specific exposure than decade-long exposure to structural racism. The risk is that we focus on what we can measure rather than what we should measure, akin to looking for the keys only under the lamppost because that is where the light is. There is, in our view, a need for bolder experiments that dare to test novel and transformative interventions that may one day help us to identify ways to meaningfully reduce health inequalities.

In conclusion, social epidemiologists would do well to embrace the exciting opportunities that experiments have to offer to understand how interventions that generate changes in social exposures lead to changes in health. Social experiments might be best understood as one piece of the arc we are trying to build with robust evidence of the social determinants of health. We should also be aware of the challenges of experiments. For RCTs, critical challenges include identifying the right level, timing, and etiological period of intervention; understanding the larger social context; and being clear about the counterfactual. For natural experiments, we highlight the challenge of reconciling results from observational studies and natural experiments and the limitations of looking at historical policy changes in particular contexts. For all experiments, we also emphasize the critical importance of interdisciplinary collaboration. Overall, experiments offer a critical tool that, together with existing theoretical and empirical knowledge, helps us to advance our understanding of the complex causal mechanisms linking social determinants and health.

ACKNOWLEDGMENTS

Author affiliations: Harvard Center for Population and Development Studies, Cambridge, Massachusetts, United States (Lisa F. Berkman); Department of Social and Behavioral Sciences, T. H. Chan School of Public Health, Cambridge, Massachusetts, United States (Lisa F. Berkman); Center for Primary Care and Public Health (Unisanté), Department of Epidemiology and Health Systems, University of Lausanne, Lausanne, Switzerland (Mauricio Avendano); and Department of Public Health, Environments and Society, London School of Hygiene and Tropical Medicine, London, United Kingdom (Emilie Courtin).

This work was supported by the Medical Research Council (MR/T032499/1).

We thank anonymous reviewers for their helpful comments.

Conflict of interest: none declared.