Objective: To provide examples of post-hoc probing of significant moderator and mediator effects in research on children with pediatric conditions.
Methods: To demonstrate post-hoc probing of moderational effects, significant two-way interaction effects (dichotomous variable × continuous variable; continuous variable × continuous variable) were probed with regressions that included conditional moderator variables. Regression lines were plotted based on the resulting regression equations that included simple slopes and y-intercepts. To demonstrate probing of mediational effects, the significance of the indirect effect was tested (i.e., the drop in the total predictor → outcome effect when the mediator is included in the model), using Sobel's (1988) equation for computing the standard error of the indirect effect.
Results: All significant moderator and mediator effects are presented in figure form.
Conclusions: The computational examples demonstrate the importance of conducting post-hoc probes of moderational and mediational effects.
The purpose of this discussion is to provide examples of post-hoc probing of significant moderator and mediator effects with data from a study of children with a pediatric condition.1 A moderator is a variable that specifies conditions under which a given predictor is related to an outcome. That is, the nature of the predictor → outcome association can vary as a function of the moderator. A mediator is a variable that serves to explain the process or mechanism by which a predictor significantly affects an outcome, such that the predictor is associated with the mediator, which is, in turn, associated with the outcome.
Although the “moderator” examples in this article demonstrate post-hoc probes, the “mediator” examples demonstrate tests of whether or not the mediational effect is significant. Thus, in the case of mediation, it is not entirely accurate to refer to such tests as “ post-hoc probes.” Instead, they could be considered a critical step in testing the significance of a mediational effect (in the same way that examining the significance of an interaction effect is a test, rather than a post-hoc probe, of a moderated effect). On the other hand, I will continue to use the phrase “post-hoc probe,” since the statistical strategies discussed here constitute a set of critical statistical tests that should be conducted above and beyond (and after) tests that are typically conducted when examining mediator and moderator effects.
This article should be considered a companion to an earlier article (Holmbeck, 1997) that included a detailed overview of terminological, conceptual, and statistical problems in the study of moderators and mediators (primarily in the pediatric literature; also see Baron & Kenny, 1986). In the earlier article, I explained how moderators and mediators are tested statistically (with regressions and structural equation modeling), but I did not discuss in any detail how one would “ probe” a significant moderator or mediator effect. Although discussions of post-hoc probing of moderational effects (Aiken & West, 1991) and mediational effects (Kline, 1998; MacKinnon & Dwyer, 1993) have received some attention in the literature, an article that includes examples of both in the same discussion, within the context of pediatric research, is not available.
What is “post-hoc probing” and why is it necessary? The answer to this question varies depending on whether we are speaking of moderation or mediation. When one tests for the presence of a moderational effect with multiple regression, one examines whether an interaction between two variables (one independent variable and a moderator) is a significant predictor of an outcome variable, after controlling for the effect of the two predictors. The presence of a significant interaction tells us that there is significant moderation (i.e., that the association between the predictor and the outcome is significantly different across levels of the moderator or that the association is conditional on values of the moderator), but tells us little about the specific conditions that dictate whether the predictor is significantly related to the outcome. For example, if one were interested in whether the association between a parenting variable (e.g., father psychological control; Holmbeck, Shapera, & Hommeyer, in press) and an outcome (e.g., school grades) is moderated by group status (e.g., spina bifida vs. an able-bodied comparison sample), one would test the interaction of psychological control and group as a predictor of school grades after controlling for the parenting and group main effects. If the interaction is significant, this tells us that the slope of the regression line (i.e., simple slope) that represents the association between parenting and grades for the spina bifida sample is significantly different from the slope for the comparison sample. Unfortunately, the significance of the interaction effect does not tell us whether either of the simple slopes is significantly different from zero. In other words, we do not know, based on the initial significant interaction effect, whether the relationship between parenting and grades is significant for the spina bifida sample, the comparison sample, or both samples. Post-hoc probing of the interaction effect (via computation of the simple slopes with statistical tests) will provide us with this information. Such information also facilitates the plotting of regression lines in figure form.
With respect to mediation, one is usually interested in whether a variable “ mediates” the association between a predictor and an outcome, such that the mediator accounts for part or all of this association (see Holmbeck, 1997, for a complete explanation). To test for mediation, one examines whether the following are significant: (1) the association between the predictor and the outcome, (2) the association between the predictor and the mediator, and (3) the association between the mediator and the outcome, after controlling for the effect of the predictor. If all of these conditions are met, then one examines whether the predictor → outcome effect is less after controlling for the mediator. The question that arises in this type of analysis is: how much reduction in the total effect is necessary to claim the presence of mediation? In the past, some have reported whether the predictor → outcome effect drops from significance (e.g., p <.05) to nonsignificance (e.g., p >.05) after the mediator is introduced into the model. This strategy is flawed, however, because a drop from significance to nonsignificance may occur, for example, when a regression coefficient drops from.28 to.27 but may not occur when the coefficient drops from.75 to.35. In other words, it is possible that significant mediation has not occurred when the test of the predictor → outcome effect drops from significance to nonsignificance after taking the mediator into account. On the other hand, it is also possible that significant mediation has occurred, even when the statistical test of the predictor → outcome effect continues to be significant after taking the mediator into account. Clearly, a test is needed for the significance of this drop.
I begin by providing two examples of post-hoc probing of moderated effects and then continue with an example involving a mediated effect. All of the examples here are based on data from an ongoing longitudinal study of children with spina bifida. Two samples are being studied: a sample of 68 children with spina bifida and a matched sample of 68 able-bodied comparison children (8 and 9 years old at the beginning of the study). Data are collected during 3-hour home visits, during which parents and children complete questionnaires and participate in videotaped family interaction tasks. Data are also gathered from teachers (for both samples) and health professionals (only for the spina bifida sample). More details about this study are provided elsewhere (Holmbeck et al., 1997, 1998; Holmbeck, Johnson, et al., in press; Holmbeck, Shapera, et al., in press; Hommeyer, Holmbeck, Wills, & Coers, 1999; McKernon et al., 2001).
Post-Hoc Probing of Significant Moderational Effects
In this section, I provide two examples of post-hoc probing of significant moderational effects: a two-way interaction involving one dichotomous and one continuous variable and a two-way interaction involving two continuous variables. (Text describing post-hoc probing of a three-way interaction is available from the author.) A detailed presentation of post-hoc probing of significant interaction effects is presented in Aiken and West (1991); this discussion draws from their work. On the other hand, this presentation differs from Aiken and West's (1991) in three ways. First, I provide an example of a two-way interaction between a dichotomous (i.e., two-level) categorical variable and a continuous variable. An example of this type of interaction is not presented in Aiken and West (1991); instead, they provide examples of interactions among continuous variables and an example of a more complex interaction between a three-level categorical variable and a continuous variable. Second, my examples are based on data from a study of a pediatric population, thus providing examples that readers of this journal are likely to find relevant. Finally, I will attempt to explain, at a fairly basic level, the rationale behind some of the necessary computations (i.e., the explanation of computations that underlie post-hoc probing of moderator effects covers less than two pages in Aiken & West's 1991 volume; pp. 18-19).
Computational Example 1
The first example of post-hoc probing involves a two-way interaction and is based on an analysis of parent and teacher questionnaire data (see Holmbeck, Shapera, et al., in press). The purpose of the overall set of analyses was to examine whether associations between three parenting variables (acceptance, behavioral control, and psychological control) and child adjustment (internalizing and externalizing symptoms, observed adaptive behavior, and school grades) were moderated by gender and group (spina bifida vs. able-bodied). All regressions were run separately for each parenting variable and for each parent gender. Seven effects were tested in each regression: three main effects (child gender, group, one parenting variable), all possible two-way interactions (Gender × Group, Gender × Parenting, Group × Parenting), and the Gender × Group × Parenting three-way interaction.2 Typically, all continuous predictor variables (including the moderator) are centered prior to conducting such regression analyses. Centering is accomplished by subtracting the sample mean from all individuals' scores on the variable, thus producing a revised sample mean of 0. This procedure reduces the multicollinearity between predictors and any interaction terms among them and facilitates the testing of simple slopes (as will be demonstrated). It does not alter the significance of the interaction, nor does it alter the values of the simple slopes. Although dichotomous variables can also be centered, interpretation is simplified by using a 0 versus 1 coding scheme, since investigators are usually interested in generating regression lines for specific groups rather than for weighted values of the moderator. In this example, two of the three main effects (i.e., gender, group) were dichotomous variables; thus, only the parenting variable was centered. If one is examining the impact of a continuous moderator, centering such a variable allows one to generate slopes (representing associations between predictor and outcome) for values ±1 SD from the mean of the moderator (see computational example 2 below; also see Aiken & West, 1991, pp. 14-22, for an example involving a continuous moderator). The “±1 SD” is merely a convention (e.g., Cohen & Cohen, 1983); other values, if theoretically meaningful, could be used instead.
Covariates may also be included in the model. These terms (i.e., regression weight × covariate) should be included in the conditional moderational equations if they are also included in the original regression equation. The grand mean value of the covariate can be substituted, which is multiplied times the regression weight for the covariate (Holmbeck, 1997). For example, if age is used as a covariate, the product of the regression weight for age (e.g.,.6721) and the grand mean for age for the entire sample (e.g., 8.71) can be included in all conditional moderator equations (which essentially produces an adjustment to the intercept term). If this term is included in the regressions, but is not taken into account when plotting the figures, the figures will “appear” correct, but all predicted values for the outcome will be off by some constant (e.g.,.6721 × 8.71). Such an adjustment for covariates assumes, of course, that associations between the covariate and the outcome are constant across levels of the predictors (i.e., homogeneity of regression).
The example that I will present first involves a significant interaction between father-reported psychological control (variable name = FPC) and group (GROUP) in predicting teacher-reported school grades (TGRADE). FPC was centered by subtracting the grand mean (based on the total sample, including the participants with spina bifida and the able-bodied participants) from each participant's score on this variable (i.e., FPC [centered] = FPC - 1.80). The two-way interaction (GRP_FPC) emerged as significant in the initial regression and remained significant in a reduced model that included only the two main effects and the interaction (i.e., after the full model was run, the nonsignificant three-way and all nonsignificant two-way interaction terms were dropped and a reduced model was run). To conduct a probe of the significant interaction, one first needs to compute two new conditional moderator variables and then run two regressions by incorporating each of these new variables (Aiken & West, 1991). Specifically, one computes conditional moderator variables where one of the groups is assigned a value of 0 in one analysis and the other group is assigned a value of 0 in the other analysis. With such a strategy, we are manipulating the 0 point of the moderator to examine conditional effects of the predictor on the outcome. I will say more about this point later.
Initially, the moderator (GROUP) was coded as 0 for the spina bifida sample and 1 for the able-bodied sample. Thus, the following compute commands (from SPSS) were employed (i.e., two new variables, GROUPSB and GROUPAB, are created; SPSS printouts for all examples in this article are available from the author):
Significance tests (t) for each slope are also provided, which indicate that the simple slope for the spina bifida sample was significant. The direction indicates that grades tend to be lower at higher levels of paternal psychological control for this sample. In a computer printout, this t value will be the significance test of the FPC variable (with both main effects and the interaction in the model). The regression lines can then be plotted by substituting high (1 SD above the mean;.27) and low (1 SD below the mean; -.27) values of FPC (centered). These lines were plotted and appear in Figure 1.
Computational Example 2
The second example of post-hoc probing involves a two-way interaction of two continuous variables and is based on an analysis of observational data (as predictors) and teacher-report grades (as an outcome). The data come from the same study already described. The purpose of the overall set of analyses was to examine whether maternal and paternal parenting variables have additive and/or interactive effects on child adjustment. This example examines observers' ratings of maternal (MBC) and paternal (FBC) behavioral control in relation to teacher-reported grades (TGRADE). Seven effects were tested in the original regression: three main effects (GROUP, MBC, and FBC), all possible two-way interactions (GROUP × MBC, GROUP × FBC, MBC × FBC), and the GROUP × MBC × FBC three-way interaction. GROUP was coded as 0 for the spina bifida sample and 1 for the able-bodied sample. MBC and FBC were centered by subtracting the grand mean from the value for each participant (i.e., MBC [centered] = MBC - 4.29; FBC [centered] = FBC - 4.07).
The two-way interaction of the parenting variables (MBC × FBC) emerged as significant in the initial regression and remained significant in a reduced model that included only the two main effects and the interaction. To conduct a probe of this significant interaction, one again needs to compute two new conditional moderator variables (Aiken & West, 1991). We assumed that MBC was the moderator. Thus, conditional moderator variables were computed as follows:
Consequences of Not Conducting Post-Hoc Probes of Moderational Effects
If an investigator is examining moderational effects by testing the significance of interaction terms, he or she likely has hypothesized previously that the impact of a predictor on an outcome is conditional on the level of a moderator variable. Suppose one has predicted that “A” will be related to “B” for males, but not for females. A significant “A × Gender” interaction effect only tells you that “A” is related to “B” differentially as a function of gender; unfortunately, this statistical test does not answer the research question of interest. Only the post-hoc probing procedure will tell you if “A” is significantly associated with “B” for males, but not for females. In the past, some investigators (including me!; see Fuhrman & Holmbeck, 1995) have merely plotted significant interaction effects and interpreted the significance of regression line slopes based on visual inspection, without conducting post-hoc probes. This strategy is likely to lead to false-positive results; I suspect that one is more likely to conclude that a slope is significantly different from 0 based on “ eye-balling” than via statistical tests. Thus, post-hoc probing is a critical step in the evaluation of a moderator effect.
One might also be tempted to employ post-hoc probing strategies that differ from those suggested here. For example, if one has isolated a significant interaction effect between a dichotomous variable and a continuous variable (see example 1), one might choose to examine the bivariate correlation between the continuous predictor and the outcome at each level of the dichotomous moderator. Similarly, if one had found an interaction between two continuous variables (see example 2), one might be tempted to examine the bivariate correlation between one of the continuous predictors and the outcome at high and low levels (usually based on a median split) of the other continuous variable. Although this bivariate correlation approach is superior to doing no post-hoc probing, this strategy is less desirable for several reasons. First, the bivariate correlation strategy does not provide the investigator with a regression equation. Without such an equation, the plotting of findings is not a straight-forward task. Second, when one generates regression line equations with slopes and intercepts, the slope is in the same metric as the outcome. Given the slope, one is able to determine the increase (or decrease) that will occur in the value of the outcome as a function of a 1 unit increase (or decrease) in the predictor (at a particular level of the moderator). Third, by computing the regression equations, one can determine mathematically where the regression lines cross (see Aiken & West, 1991, pp. 23-24), which may be of practical or theoretical interest. Finally, the post-hoc strategy discussed here allows for greater flexibility in computing and plotting regression lines. In the case of an interaction between two continuous variables, one can use the ±1 SD convention or a variety of other values. When using the bivariate correlation strategy, one typically uses only the median split approach. An additional drawback of the median split strategy is that it yields a correlation for a fairly diverse subsample of participants (i.e., the association between the predictor and outcome for all individuals above or below the median on the continuous moderator). The post-hoc strategy discussed in this article allows one to examine associations between predictor and outcome at any possible values of the moderator.
Post-Hoc Probing of Significant Mediational Effects
When one has satisfied the conditions of mediation, as described earlier, one can test the significance of the indirect effect, which is mathematically equivalent to a test of whether the drop in the total effect (i.e., the zero-order predictor → outcome path) is significant upon inclusion of the mediator in the model. This mathematical relationship is demonstrated in the following (see MacKinnon & Dwyer, 1993):
In this case, the direct effect is the predictor → outcome path with the mediator already in the model. Thus, the significance test of the indirect effect is equivalent to a significance test of the difference between the total and direct effects, with the latter representing the drop in the total effect after the mediator is in the model. The indirect effect is the product of the predictor → mediator and mediator → outcome path coefficients (the latter path coefficient is computed with the predictor in the model; Cohen & Cohen, 1983).
To conduct the statistical test for mediation, one needs unstandardized path coefficients from the model, as well as standard errors for these coefficients (all available in computer output). One also needs the standard error of the indirect effect. Sobel (1988; also see Baron & Kenny, 1986; Kline, 1998) presents an equation for computing the standard error of the indirect effect, as follows:
MacKinnon has argued recently that the Sobel equation may be overly conservative, with low power and inaccurate Type I error rates (see David MacKinnon's web sites: www.public.asu.edu/∼davidpm/ripl/david_mackinnon.htm or www.public.asu.edu/∼davidpm/ripl/mediate.htm). He and others are currently considering alternative approaches to testing the significance of indirect effects (also see David Kenny's web site: nw3.nai.net/∼dakenny/mediate.htm).
An interactive web site is available that conducts the Sobel test (with significance tests) if path coefficients and standard errors are entered (http://quantrm2.psy.ohio-state.edu/kris/sobel/sobel.htm).
The following example is based on data from the same study of children with spina bifida, discussed earlier. Specifically, the data reported here are presented in Holmbeck, Johnson, et al. (in press). As can be seen in Figure 3, we tested a model where parents' willingness to grant autonomy (a) to their offspring was viewed as a mediator of associations between maternal overprotectiveness (o) and externalizing symptoms (e). The model is abbreviated as follows:
In conducting regressions for mediational analyses, it is suggested that the same n be used for all analyses. If ns vary across regressions, computational anomalies are possible (e.g., the total effect may not equal the sum of the indirect and direct effects).
Figure 3 illustrates the mediational effect. Values on paths are path coefficients (standardized β s). Although unstandardized bs are used in the calculations discussed above, standardized βs are often included in figures of mediated effects. Path coefficients outside parentheses are zero-order correlations (rs). Path coefficients in parentheses are standardized partial regression coefficients from equations that include the other variable with a direct effect on the criterion.
Consequences of Not Testing the Significance of a Mediated Effect
As was the case with moderated effects, failure to test the significance of a mediated effect is likely to lead to false conclusions. Although failure to probe moderated effects is likely to result in false-positive conclusions, both false-positive and false-negative conclusions are possible when one fails to test the significance of a mediated effect. Suppose one tests the utility of a mediational model and seeks to determine whether the significance of the total effect drops to nonsignificance after the mediator is taken into account. Also suppose that this “drop to non-significance” criterion is used as the basis for whether significant mediation has taken place. If an initial total effect was just below the p <.05 threshold of significance (e.g., p =.049) and then dropped so that the significance level was now just above the p <.05 threshold (e.g., p =.061), one might conclude that significant mediation has occurred. Upon further analysis (using the strategy employed here), however, one may find that this represents a false-positive conclusion. On the other hand, if the original total effect was well under the significance threshold (e.g., p =.001) and remained under the threshold (e.g., p =.040) after accounting for the mediator, one might conclude that no mediation occurred (which would likely be a false-negative conclusion). Indeed, analyses using the Sobel equation may reveal that significant mediation had occurred in this latter case. As noted earlier, the “drop to nonsignificance” criterion is flawed. There may be other reasons for false-negative conclusions (i.e., Type II errors). For example, there may be limited power to detect an effect, or the measures may be weak (e.g., they may have low reliability).
The purpose of the computational examples included here was to demonstrate the importance of conducting post-hoc probes of moderational and mediational effects in studies of pediatric populations. This article can be used in conjunction with the earlier Holmbeck (1997) article on moderator and mediator effects. To demonstrate post-hoc probing of moderational effects, significant two-way interaction effects were probed with regressions that included conditional moderator variables. Regression lines were plotted based on the resulting regression equations. To demonstrate probing of mediational effects, the significance of the indirect effect was tested (i.e., the drop in the total predictor → outcome effect when the mediator is included in the model), making use of Sobel's (1988) equation. Such data analytic strategies should prove useful for investigators seeking to examine the utility of prediction models that include hypothesized mediator and moderator variables (e.g., Thompson & Gustafson, 1996; Wallander & Varni, 1998).
Completion of this work was supported by a grant (12-FY01-0098) from the March of Dimes Birth Defects Foundation. I thank Craig Colder for his comments on an earlier draft of this article.