Improving the Efficiency of Outbound CATI As a Nonresponse Follow-Up Mode in Address-Based Samples: A Quasi-Experimental Evaluation of a Dynamic Adaptive Design

Abstract

This article evaluates the use of dynamic adaptive design methods to target outbound computer-assisted telephone interviewing (CATI) in the California Health Interview Survey (CHIS). CHIS is a large-scale, annual study that uses an address-based sample (ABS) with push-to-Web mailings, followed by outbound CATI follow-up for addresses with appended phone numbers. CHIS 2022 implemented a dynamic adaptive design in which predictive models were used to end dialing early for some cases. For addresses that received outbound CATI follow-up, dialing was paused after three calls. A response propensity (RP) model was applied to predict the probability that the address would respond to continued dialing, based on the outcomes of the first three calls. Low-RP addresses were permanently retired with no additional dialing, while the rest continued through six or more attempts. We use a difference-in-difference design to evaluate the effect of the adaptive design on calling effort, completion rates, and the demographic composition of respondents. We find that the adaptive design reduced the mean number of calls per sampled unit by about 14 percent (relative to a modeled no-adaptive-design counterfactual) with a minimal reduction in the completion rate and no strong evidence of changes in the prevalence of target demographics. This suggests that RP modeling can meaningfully distinguish between ABS sample units for which additional dialing is and is not productive, helping to control outbound dialing costs without compromising sample representativeness.

Statement of Significance

Outbound CATI using appended telephone numbers is a common secondary contact mode in push-to-Web address-based sample (ABS) studies. However, dialing and interviewing can add substantial costs, raising the question of whether more granular targeting of outbound dialing is possible. We demonstrate that, when CATI is used for follow-up in a push-to-Web ABS study, a dynamic adaptive survey design can reduce dialing effort while maintaining completion rates and the demographic representativeness of the responding sample. This design relies on predictive models to end dialing early for cases for which further dialing is unlikely to be productive.

To our knowledge, this is the first published evaluation of the use of dynamic adaptive design methods to target outbound CATI in a mixed-mode ABS study. Such designs can be considered a cost-reducing strategy in collections that are sufficiently large and/or recurring, such that the reduction in variable dialing costs outweighs the (relatively modest) fixed set-up costs. Additionally, we demonstrate how quasi-experimental methods can be used to evaluate a survey design change when a randomized experiment is not feasible. In this case, we use a difference-in-difference strategy, exploiting the availability of a recent prior administration of the same study using otherwise similar data collection procedures.

1. INTRODUCTION

ABS designs, in which residential addresses are selected from the US Postal Service’s Computerized Delivery Sequence (CDS) file and contacted by mail, have become common in surveys of US households (Harter et al. 2016; Olson et al. 2019). “Push-to-Web” mailings, providing a URL and login credentials for a Web instrument, are well suited to ABS studies (Smyth et al 2010; Messer and Dillman 2011; Dillman 2017). However, reliance on a single mode, particularly Web, raises a risk of nonresponse bias. Stated mode preferences vary across observable subsets of the US population (Olson et al. 2012; Smyth et al. 2014), and the combination of available modes affects respondent composition (Smyth et al. 2010; Messer and Dillman 2011; Baumgardner et al. 2014; Nichols et al. 2015) and substantive estimates (Brick et al. 2022). Though only 7 percent of US adults do not use the Internet at all (Perrin and Atske 2021a), disparities in facility of access (Perrin 2021; Vogels 2021a, 2021b; Atske and Perrin 2021; Government Accountability Office 2022) and frequency of use (Perrin and Atske 2021b) may drive nonresponse biases related to the ability or willingness to respond to surveys online (Cornesse and Bosnjak 2018).

Therefore, offering multiple modes is a best practice (Dillman et al. 2014). Common secondary modes in push-to-Web ABS studies are hardcopy questionnaires and computer-assisted telephone interviewing (CATI). CATI response can be offered by providing a telephone number that respondents can call to complete the survey by phone (“inbound CATI”). Some ABS sample vendors match to external databases to append phone numbers for some sampled addresses, allowing matched sample units to be contacted by phone (“outbound CATI”).

CATI has some desirable properties as a secondary mode in push-to-Web ABS studies. First, CATI programs allow automatic routing through skip patterns. Second, CATI allows for interviewer assistance of respondents who face language or literacy barriers in completing self-administered questionnaires (Gerber and Wellens 1995; Brick et al. 2011).

A key drawback of outbound CATI is the cost of dialing and live interviewing. Once outbound CATI is introduced into a mixed-mode ABS study, researchers face cost drivers that are similar to random digit dial (RDD) samples, including low completion-to-dial rates and regulatory restrictions on autodialing of cell phone numbers. Thus, the use of outbound CATI increases the need to carefully balance data quality (particularly nonresponse bias minimization) and costs. This makes it a natural candidate for consideration as a lever in an adaptive survey design (Groves and Heeringa 2006; Chun et al. 2018).

An adaptive design targets one or more data collection features (“interventions”) to subsets of cases within a given sample, based on observable and/or predicted characteristics of those cases. Interventions are changes to data collection efforts relative to some baseline protocol. Lynn (2017), Tourangeau et al. (2017), and Schouten et al. (2018a) provide useful overviews of early adaptive designs. Recent adaptive designs have targeted incentives (Seeskin et al. 2019; Jackson et al. 2020; McGonagle et al. 2022; Peytchev et al. 2022), the number of follow-up contacts (Coffey et al. 2020), case prioritization and interviewer allocation (Gummer and Blumenstiel 2018; Tolliver et al. 2019; West et al. 2021a), location effort (Seeskin et al. 2019), priority or FedEx mailings (Kaputa et al. 2017; Johnson 2018; Seeskin et al. 2019; Medway et al. 2022), and mode offerings (see below).

Schouten et al. (2018a) distinguish between “static” and “dynamic” adaptive designs. In static designs, interventions are allocated before collection, based on auxiliary data available on (or linked to) the sample file. In dynamic designs, interventions are assigned mid-collection, factoring in paradata accumulated from the collection process itself. Adaptive designs typically involve predictive modeling to translate auxiliary data and/or paradata into case-level predictions of quantities being optimized on, such as response propensity (RP), contribution to bias, and/or the cost of obtaining a complete (Wagner and Hubbard 2014; Burger et al. 2017; Schouten et al 2018b; Wagner 2019; Olson et al. 2021; West et al. 2021b; Peytchev et al. 2022).

Several published studies have evaluated the targeting of contact or response modes as adaptive design interventions; but, to our knowledge, none have specifically evaluated the use of outbound CATI as a dynamic adaptive design intervention within a cross-sectional US ABS study. Several studies, not all using the ABS frame, have used adaptive designs to vary the timing of the introduction of a hardcopy questionnaire (Chestnut 2013; Zimmer et al. 2016; McGonagle and Freedman 2017; Freedman et al. 2018; U.S. Census Bureau 2019; Jackson et al. 2023). Coffey et al. (2020) tested a dynamic adaptive design in which CATI follow-up contacts were among the targeted interventions, using a sample drawn from American Community Survey respondents (thus with much richer auxiliary data than a typical ABS frame). Other uses of adaptive design methods to direct outbound dialing (LaFlamme and Karaganis 2010; LaFlamme and St-Jean 2011; Luiten and Schouten 2013; Lundquist and Särndal 2013; Wagner 2013) have been applied to non-ABS frames and/or in non-US contexts.

We contribute to this literature by evaluating a dynamic adaptive design targeting outbound CATI effort in the 2022 California Health Interview Survey (CHIS 2022). CHIS is an annual, large-scale study sponsored by the University of California, Los Angeles (UCLA) Center for Health Policy Research. Data are collected by SSRS. CHIS collects data related to public health and health care access. Beginning in 2019, CHIS transitioned from RDD samples to ABS samples with push-to-Web mailings, retaining outbound CATI as a secondary contact mode for nonresponse follow-up.

Shortly after the CHIS 2022 collection began, lower-than-expected Web completion rates necessitated reducing the number of outbound call attempts to control costs. A dynamic adaptive design was developed to halt dialing early for some nonresponding cases. In particular, the intervention (an early end to dialing) was applied based on models of (i) the probability that the address included members of key target subpopulations and (ii) the probability that continued dialing would obtain a response. This extends the approach of Coffey et al. (2020)—continual monitoring to allocate an intervention in “real time” as cases reach the intervention point—to an ABS setting.

To meet the required cost reduction, the CHIS 2022 adaptive design was applied nonexperimentally. However, the availability of a recent prior collection that used otherwise similar field procedures creates a plausible quasi-experimental design. Using a difference-in-differences (DiD) approach, we evaluate the extent to which the CHIS 2022 adaptive design (i) reduced data collection costs (proxied by dialing volumes) while (ii) maintaining a constant overall completion rate and (iii) maintaining or improving the prevalence of key target subpopulations among respondents.

2. DATA AND METHODOLOGY

This section summarizes the CHIS sample design and data collection procedures, the CHIS 2022 adaptive design, and the quasi-experimental evaluation methodology. Additional methodological details are available in the supplementary data online. Full technical documentation and public-use data for CHIS are available at UCLA Center for Health Policy Research (2022).

2.1 Baseline CHIS 2021 and 2022 Design

CHIS is fielded annually, and a pooled dataset is released every 2 years combining an odd and even year’s samples. Consequently, the sample design and data collection procedures are largely held constant across the two pooled years. CHIS 2021 and 2022 used a multi-frame design blending a stratified ABS sample with an RDD oversample of prepaid cell phones. All analysis reported in this paper is from the ABS sample and, therefore, the following methodological details pertain only to the ABS sample. Unless otherwise noted, all methodological details apply to both CHIS 2021 and CHIS 2022.

2.1.1 Sample design

The CHIS ABS samples were sourced from the CDS file via Marketing Systems Group (MSG), an ABS frame vendor. The target population was the noninstitutionalized household population in California. Stratification was used to meet overlapping sample size targets for (i) small geographic areas and (ii) demographic subgroups of analytic interest to the CHIS sponsor and data users. Geographic strata were primarily defined by counties (combining some small counties and subdividing some larger counties). These geostrata were crossed with “modeled strata” defined by the probability that an address included a member of a target demographic subgroup. Table 1 lists the modeled strata used for CHIS 2021 and 2022. Some modeled strata were assigned using preexisting MSG flags, while others were based on the density of the target subgroup at the Census block group level. Others were assigned using custom models developed by the CHIS data collection team. These custom models consisted of random forests (Breiman 2001) built on prior CHIS waves. Predictors were demographic and behavioral variables matched to the sample file at the address level by a commercial data vendor. Two residual strata captured addresses that did not match to the external databases and/or were not modeled to include one of the target characteristics. Additional information about these strata is provided in the supplementary data online.

Table 1.

CHIS 2021 and 2022 Modeled Strata Definitions

Stratum name	Definition
Vietnamese	Vendor flagged as Vietnamese surname
Korean	Vendor flagged as Korean surname
Asian language	Modeled likely Asian-language interview or geographies with high density of Asian language speakers
Spanish language	Modeled likely Spanish-language interview or geographies with high density of Spanish speakers
Hispanic	Vendor flagged as Hispanic surname
Other non-English	Geographies with high density of non-English speakers
Other Asian	Modeled likely Asian-American or vendor flagged as Asian surname
African American	Geographies with high density of African Americans
With children	Modeled likely household with children
Other 65+	Modeled or vendor flagged likely household with 65+ adult
Residual—match	All other addresses matched to vendor voter or consumer file
Residual—no match	All other addresses not matched to vendor voter or consumer file

Note.—Strata were assigned hierarchically in the order shown here; for example, an address that is both flagged as Korean surname and modeled as a likely Asian-language interview would be assigned to the Korean stratum.

2.1.2 Data collection

CHIS 2021 and 2022 used a mixed-mode mailed push-to-Web design. The CHIS 2021 and CHIS 2022 samples were each selected and released in 25 waves staggered throughout the year, with 1–2 weeks between the release of each wave. For CHIS 2021, the first wave was released in March 2021 with interviews accepted through November 30; for CHIS 2022, the first wave was released in February 2022 and interviews accepted through November 1. There were no wave-specific end dates: interviews were accepted from all waves through the end of the year’s collection. The supplementary data online provides the exact dates of each mailing, and of the start of outbound dialing, for each wave.

Figure 1 illustrates the contact protocol for a single wave. Contact began with a mailed invitation that included a $2 noncontingent incentive. The invitation provided a URL and passcode for the Web survey instrument, as well as a toll-free number for inbound CATI interviews. Nonrespondents received up to three follow-up mailings: a sealed postcard, a reminder letter, and another sealed postcard. Within each wave, each mailing was sent approximately 2 weeks after the previous mailing. All follow-up mailings offered both the Web and inbound CATI response options. Both Web and CATI interviews could be completed in English, Spanish, Chinese, Vietnamese, Korean, and Tagalog. The invitation letters included text in all six languages; for some modeled strata, bilingual text with the applicable non-English language was featured on the envelope.

Figure 1.

Contact Protocol for a Typical CHIS 2021–2022 Wave. Timeframes are approximate and varied slightly depending on the wave. For exact wave-specific mailing and outbound CATI dates, see the supplementary data online.

Approximately 2 weeks after the final mailing, outbound CATI began for nonresponding addresses to which the sample vendor matched one or more cell and/or landline phone numbers. As initially planned, all addresses could receive up to six outbound calls. In practice, toward the end of both years, some strata received additional dialing attempts to meet sample size targets. In CHIS 2022 (but not 2021), secondary phone numbers were used where available: the call center switched to the secondary number after 1–3 dialing attempts, following rules described in the supplementary data online. There were no further mailings once outbound CATI began, but Web responses continued to be accepted.

All responding households were asked to complete an “adult” interview in which one person in the household aged 18 or older, chosen using the next-birthday method (Salmon and Nichols 1983), reported about themselves. Proxy responses were allowed for adults unable to respond for themselves due to chronic illness or disability status. No postincentive was offered for completion of the adult interview. The average adult interview length was 72 minutes for CATI and 47 minutes for the Web instrument. Households with children under 18 were also asked to complete child and/or adolescent interviews. However, the analysis reported in this paper focuses on adult interview completion as the response outcome of interest. The adult interview response rate using the American Association for Public Opinion Research (AAPOR) response rate 4 was 8.9 percent in 2021 and 8.3 percent in 2022.

2.2 CHIS 2022 Adaptive Design

In CHIS 2021, all sampled addresses that reached the CATI phase and had an available phone number could receive up to six (and sometimes more) dialing attempts. The CHIS 2022 adaptive design limited some addresses to three outbound dialing attempts. The implementation of the adaptive design began on May 24, 2022, and continued through the end of the field period.

This strategy was operationalized using a random forest RP model predicting the probability that a case would respond with continued dialing, conditional on not having responded after three calls. The RP model was trained on the subset of the CHIS 2021 sample with at least four calls. The dependent variable was an indicator of eventual adult interview completion. The predictors included:

• Paradata from the first three calls, namely, each call’s duration and detailed operational outcome code (see the supplementary data online for categories).

• Static predictors: the address’s modeled stratum; whether the primary phone number was a landline or cell phone; and whether a second phone number was available.

The supplementary data online provides additional information about the RP model, including model fit statistics.

For each CHIS 2022 address that passed the third call without responding, dialing was temporarily paused. Three times per week, we used the RP model to assign a predicted RP score to all paused cases. For cases with a score below a predetermined threshold (with exceptions detailed in the supplementary data online), dialing ceased for the remainder of the field period. For those above the stopping threshold, dialing resumed.

The RP thresholds for intervention were based on percentiles of the RP scores. The adaptive design incorporated demographic representativeness by varying the stopping threshold across the modeled strata. These thresholds were chosen based on simulations run on CHIS 2021 data as part of the model development process. We used a lower threshold (stopping fewer cases) for strata in which this simulation suggested a higher risk of not meeting sample size targets. The threshold was set at the 50th percentile for the Vietnamese, Korean, Asian language, Hispanic, and Other Asian strata; the 40th percentile for the Spanish Language stratum; and the 10th percentile for the Other Non-English, African American, and With Children strata. For the two Residual strata, as well as one stratum for which targets have historically been exceeded in CHIS (“Other 65+”), the threshold was set at the 100th percentile; that is, all cases in these strata (with the exceptions detailed in the supplementary data online) were stopped after three calls.

Implicitly, therefore, the adaptive design used two sets of models: the RP model predicting the probability of responding to further dialing, with intervention more likely for lower-RP addresses; and the stratification models predicting the probability that (conditional on response) the address would yield a response from a member of a target subgroup, with intervention less likely for addresses more likely to include target subgroups.

2.3 Analytic Approach: DiD

To maximize the cost reductions, the adaptive design was not implemented as a randomized experiment. Therefore, we use quasi-experimental difference-in-difference (DiD) analyses (Card and Krueger 1994; Angrist and Pischke 2009) to assess its effect on survey completion rates, the demographic composition of the responding sample, and calling effort. The canonical DiD framework assumes two groups (“nontreated” and “treated”) and two periods: “pretreatment” in which neither group receives the treatment, and “posttreatment” in which the treated group receives the treatment. The DiD estimator of the treatment effect on the mean of outcome y is:

$$\mathrm{ }\delta =\left({\stackrel{-}{y}}_{\mathrm{treated},\mathrm{post}}-\mathrm{ }{\stackrel{-}{y}}_{\mathrm{treated},\mathrm{pre}}\right)-\left({\stackrel{-}{y}}_{\mathrm{non}-\mathrm{treated},\mathrm{post}}-{\stackrel{-}{y}}_{\mathrm{non}-\mathrm{treated},\mathrm{pre}}\right).$$

(1)

A causal interpretation of $\delta$ rests on the parallel trends assumption: in the absence of the treatment, the difference between the pre- and posttreatment means would have been the same between the treated and nontreated groups, though the pretreatment means need not have been the same. This assumption is nonverifiable, but its plausibility can be assessed by comparing pretreatment trends between the treated and nontreated groups. Extensions for multiple periods and/or multiple groups are parameterized using fixed-effects regressions (Angrist and Pischke 2009). Aside from the adaptive design, CHIS 2021 and 2022 used similar data collection procedures, making the parallel trends assumption plausible (see the supplementary data online for other differences between the collections). We empirically assess the reasonableness of this assumption for CHIS as part of the discussion of results.

For this analysis, we classify the 25 CHIS 2021 and 25 CHIS 2022 waves into the following discrete treatment groups:

• Untreated, comprising all 25 CHIS 2021 waves, for which the adaptive design was never implemented.

• Partially treated, comprising the first eight CHIS 2022 waves, for which outbound dialing began before the first implementation of the adaptive design (May 24). For these, open cases with three or more calls as of May 24 were included in the first round of RP scoring and (potential) intervention. We call these “partially” treated because many cases had more than three calls (and some reached six calls) before the adaptive design began, reducing their “exposure” to the adaptive design.

• Fully treated, comprising the remaining 17 CHIS 2022 waves, for which outbound dialing began after May 24 and which therefore were exposed to the full adaptive design protocol.

As detailed in the supplementary data online, several alternative codings of treatment exposure yield similar results as this coding.

Our primary outcomes of interest are:

• The unit-level adult interview completion rate, defined as the proportion of all sampled addresses from which a complete or partial adult interview was obtained. We focus on the adult interview because, unlike the teen or child interviews, it was attempted for all responding addresses. This outcome is a simple operational yield rather than an AAPOR response rate, as ineligible addresses are retained in the denominator.

• The percentage of adult interview completes who are members of key target demographic and other subgroups—Hispanic or African American adults, those ages 18–29 or 65+, those without a high school diploma, those with incomes below 200 percent of the federal poverty line, those living in households with children, and responses in Spanish or an Asian language. These are the subgroups for which the sponsor requested that the adaptive design be optimized to maintain their prevalence among respondents, as they are of high analytic interest to CHIS data users. Additionally, as shown in the supplementary data online, substantive CHIS estimates differ between these subgroups and the general population, meaning that avoiding underrepresentation of these groups should help to control nonresponse bias.

• The mean number of calls per sampled address, our proxy for variable cost reductions associated with the adaptive design.

We use two variants of a DiD strategy, exploiting different sources of within-sample variability, which we call the wave-dial and the wave-cohort approaches. For both, we present descriptive estimates and visualizations using variations of (1) and use fixed-effects regressions to test for statistically significant effects. As described below, the regression specifications vary by approach and by outcome.

2.3.1 Wave-dial approach

For the wave-dial approach, we define the pre- and posttreatment “periods” in terms of the number of calls received. We exploit the fact that the intervention (stopping dialing for low-RP cases) was applied only after at least three calls, and therefore did not affect any outcomes as of call 3. Across the treatment groups, we compare the difference between (i) an outcome that includes respondents with 4+ calls and (ii) the outcome limited to respondents with 0–3 calls. Formally, we estimate the effect of the adaptive design on the mean of an outcome y within the fully treated group as:

$${\delta }_{\mathrm{fully}\mathrm{\hspace{0.17em}}\mathrm{treated}}=\left({\stackrel{-}{y}}_{\mathrm{fully}\mathrm{\hspace{0.17em}}\mathrm{treated},\mathrm{all}}-{\stackrel{-}{y}}_{\mathrm{fully}\mathrm{\hspace{0.17em}}\mathrm{treated},0-3}\right)-\left({\stackrel{-}{y}}_{\mathrm{untreated},\mathrm{all}}-{\stackrel{-}{y}}_{\mathrm{untreated},0-3}\right),$$

(2)

where ${\stackrel{-}{y}}_{\mathrm{all}}$ is the mean including all respondents (including those with 4+ calls) while ${\stackrel{-}{y}}_{0-3}$ is the mean including only respondents with three or fewer calls. We use an analogous formula for the partially treated group. The parallel trends assumption requires that, in the absence of the adaptive design, the incremental effect of calls 4 and beyond would have been the same between CHIS 2021 and 2022. We compare the incremental effects of calls 1–3 (relative to no calls) to assess this assumption’s plausibility: since calls 1–3 were unaffected by the adaptive design, their incremental effect should be similar between the collections.

To test the wave-dial-estimated treatment effect on the completion rate, we estimate a panel regression. We create a panel of cumulative completion rates by wave and “call set.” Let $C_{w,d}$ be the cumulative base-weighted completion rate for wave w as of call set d, where d = 0, 1–3, or 4+. For example, $C_{w,0}$ is the completion rate for wave w excluding respondents who received any calls, while $C_{w,4+}$ is the final completion rate with all calls included. This panel has 141 observations—47 waves by 3 call sets (treating waves released on the same day as a single wave). We estimate the following weighted least squares (WLS) regression on this panel, weighting each observation by its wave’s sample size:

$${\mathrm{\hspace{0.17em}}C}_{w,d}=\mathrm{\hspace{0.17em}}{\gamma }_w+{\gamma }_d+{\delta }_{\mathrm{partial}}{T}_{\mathrm{partial}}+{\delta }_{\mathrm{full}}{T}_{\mathrm{full}},$$

(3)

where

• ${\gamma }_w$ denotes wave fixed effects.

• ${\gamma }_d$ denotes call set fixed effects.

• $T_{\mathrm{partial}}$ is an interaction term: 1 if w is a wave in the partially treated group and d is equal to 4+ dials, and 0 otherwise.

• $T_{\mathrm{full}}$ is an analogous interaction for the fully treated group.

The interaction coefficients ${\delta }_{\mathrm{partial}}$ and ${\delta }_{\mathrm{full}}$ estimate the DiD treatment effects for the partially and fully treated groups, respectively. A panel regression is needed to test the effect on completion rates because a sample record’s completion status is time-variant with respect to one of the regressors (call set, the implicit measure of “time” in this panel). That is, individual-level completion status is not static with respect to the number of dials, but rather can change as the number of calls increases. With the wave-dial analysis of completion rates, we aim to estimate (i) how the cumulative number of calls affects a case’s cumulative probability of having responded (captured by the call set fixed effects, ${\gamma }_d$⁠) and (ii) whether the effect of the last set of calls depends on whether the case was exposed to the adaptive design (captured by the interaction coefficients ${\delta }_{\mathrm{partial}}$ and ${\delta }_{\mathrm{full}}$⁠). These effects could be estimated using a sample-member-by-dial panel with cumulative unit-level completion status as the dependent variable; for computational efficiency, we collapse to a wave-by-dial panel with the cumulative response rate as the dependent variable. We use WLS, weighting by sample size, to obtain equivalent coefficients as a sample-member-by-dial regression. Because cumulative completion rates are serially correlated within waves, we cluster standard errors at the wave level (Bertrand et al. 2004).

We use a different regression specification to test the wave-dial-estimated effect on respondent composition. For each subgroup of interest, we estimate the following cross-sectional, unit-level logistic regression on the set of survey respondents:

$$\ln \left(\frac{s_i}{1-s_i}\right)=\mathrm{\hspace{0.17em}}{\gamma }_{i,w}+{\gamma }_{i,d}+{\delta }_{\mathrm{partial}}{T}_{\mathrm{partial}}+{\delta }_{\mathrm{full}}{T}_{\mathrm{full}},$$

(4)

where

• $s_i$ is the probability that respondent i is in the specified subgroup.

• ${\gamma }_{i,w}$ denotes fixed effects for respondent i’s wave w.

• ${\gamma }_{i,d}$ denotes fixed effects for the number of calls d (0, 1–3, or 4+) recorded for respondent i prior to response.

• $T_{\mathrm{partial}}$ and $T_{\mathrm{full}}$ are DiD interaction terms defined analogously to (3).

Because this regression is estimated on respondents, the implicit panel structure does not exist: the individual-level outcome (whether a respondent is a member of the specified subgroup) is static with respect to the number of calls, and the number of calls received prior to response is a fixed characteristic conditional on response. Accordingly, a cross-sectional regression specification allows us to estimate (i) how the prevalence of the subgroup differs between respondents who received no calls, 1–3 calls, or 4+ calls (captured by ${\gamma }_{i,d}$⁠) and (ii) whether the difference in prevalence in the 4+ group depends on exposure to the adaptive design (captured by ${\delta }_{\mathrm{partial}}$ and ${\delta }_{\mathrm{full}}$⁠).

2.3.2 Wave-cohort approach

For the wave-cohort approach, we depart from periodization and instead rely on the existence of two “intervention cohorts:” a low-RP cohort (cases with RP scores below the threshold for their stratum, stopped after call 3 in CHIS 2022) and a high-RP cohort (cases with RP scores above this threshold, allowed to continue past call 3 in both years) within each treatment group. We estimate the effect of the adaptive design as:

$$\mathrm{ }{\delta }_{\mathrm{fully}\mathrm{ }\mathrm{treated}}=\left({\stackrel{-}{y}}_{\mathrm{fully}\mathrm{\hspace{0.17em}}\mathrm{treated},\mathrm{low}-\mathrm{RP}}-{\stackrel{-}{y}}_{\mathrm{fully}\mathrm{\hspace{0.17em}}\mathrm{treated},\mathrm{high}-\mathrm{RP}}\right)-\left({\stackrel{-}{y}}_{\mathrm{untreated},\mathrm{low}-\mathrm{RP}}-{\stackrel{-}{y}}_{\mathrm{untreated},\mathrm{high}-\mathrm{RP}}\right),$$

(5)

and analogously for the partially treated group. Here, we assume that, without the adaptive design, the difference between the low- and high-RP outcomes would have been constant from 2021 to 2022. Because the RP model includes paradata from the first three calls, an RP score can only be assigned to cases with at least three calls. Therefore, wave-cohort analyses are restricted to cases with at least three calls.

The RP cohort to which a case belongs is a fixed characteristic, so, unlike the wave-dial method, the wave-cohort method does not entail an implicit panel structure. Accordingly, we analyze completion rates using the following individual-level, cross-sectional regression estimated on the subset of the fielded sample with three or more calls:

$$\ln \left(\frac{p_i}{1-p_i}\right)=\mathrm{ }{\gamma }_{i,w}+\mathrm{ }{\gamma }_{i,c}+\mathrm{ }{\delta }_{\mathrm{partial}}{T}_{\mathrm{partial}}+{\delta }_{\mathrm{full}}{T}_{\mathrm{full}},$$

(6)

where

• $p_i$ is the completion probability for sampled unit i.

• ${\gamma }_{i,w}$ denotes fixed effects for unit i’s wave w.

• ${\gamma }_{i,c}$ denotes fixed effects for unit i’s intervention cohort c.

• $T_{\mathrm{partial}}$ is an interaction coded as 1 if the case is in a partially treated wave and the low-RP cohort, and 0 otherwise.

• $T_{\mathrm{full}}$ is an analogous interaction term for the fully treated group.

Again, the interaction coefficients ${\delta }_{\mathrm{partial}}$ and ${\delta }_{\mathrm{full}}$ capture the DiD-estimated treatment effect.

2.3.3 General analytic procedures

To account for sample design differences between CHIS 2021 and 2022, all results shown here are base-weighted by the inverse of the unit’s selection probability. Though individual waves of each collection were allocated similarly between strata, we calculated base weights independently by wave to correct for any incidental between-wave differences in stratum allocations.

For the RP model used in the adaptive design, the response outcome was completion of the adult interview, since the adult interview was attempted for all responding addresses. Accordingly, for the purpose of these results, “respondents” refers to adult interview completes (including partials, who comprised approximately 5 percent of respondents in both years). This includes both Web and CATI interviews, including Web responses submitted after the outbound CATI operation began.

All results shown here are from the subset of the CHIS 2021 and 2022 samples for which exactly one phone number was appended to the sample (n = 157,321 for CHIS 2021, 41,545 for CHIS 2022 partially treated, and 93,482 for CHIS 2022 fully treated). We exclude addresses without an appended phone number because they were not dialed. We exclude addresses with two appended phone numbers because CHIS 2022, unlike CHIS 2021, switched to the alternate number under certain conditions. We exclude the affected units from the analysis to avoid a potential violation of the parallel trends assumption.

For brevity, when showing the results of the DiD regressions, the tables in the main text report only the interaction coefficients (⁠${\delta }_{\mathrm{partial}}$ and ${\delta }_{\mathrm{full}}$⁠), which capture the treatment effect. Full regression tables are provided with the supplementary online materials. The supplementary data online also provides a full PRICSSA item checklist (Seidenberg et al. 2023) and the wording of all survey items used in the analysis.

3. RESULTS

3.1 Completion Rates

We first consider the effect of the adaptive design on the adult interview completion rate for CHIS 2022. Because the RP model had less-than-perfect accuracy, we expect some negative impact on the completion rate. In fact, both the wave-dial and wave-cohort approach show a negative, but small, effect on the adult interview completion rate.

3.1.1 Wave-dial approach

Figure 2 shows cumulative adult interview completion rates at three points in the collection: (i) without any dials (i.e., excluding completes with at least one call), (ii) adding completes with one to three calls, and (iii) adding the remaining completes with four or more calls. The incremental increase from calls 1–3 (unaffected by the adaptive design) is similar between the three treatment groups, supporting the parallel trends assumption. Continuing to the effect of calls 4+, there is a slight downward “bend” in the trend for CHIS 2022; under the DiD framework, this reflects the estimated effect of the adaptive design.

Figure 2.

Cumulative Adult Interview Completion Rate by Number of Calls and Treatment Group. Completion rates are base-weighted by the address’s inverse selection probability and limited to addresses with one appended phone number. Completion rates are cumulative as of the specified number of calls. Denominator is the fielded sample for the treatment group (157,321 for 2021 untreated; 41,545 for 2022 partially treated; and 93,482 for 2022 fully treated).

Table 2 reports a simple descriptive estimate of the effect of the adaptive design, using (2). It also shows the coefficients on the DiD interaction terms from the corresponding regression specification [${\delta }_{\mathrm{partial}}$ and ${\delta }_{\mathrm{full}}$ from (3)]. Descriptively, we estimate that the CHIS 2022 adaptive design decreased the final completion rate by about 0.2 percentage points in the partially treated waves and 0.3 percentage points in the fully treated waves, corresponding to a 2–3 percent decrease in relative terms. The DiD regression coefficient is statistically significant for the fully treated group.

Table 2.

Effect of Adaptive Design on CHIS Completion Rate, Wave-Dial Method

Treatment groupa	Cumulative completion rate, by calling category (%)			Incremental completion rate, by calling category (%)		Difference-in-difference regressionb
	No calls	1–3 calls	4+ calls	1–3 calls	4+ calls	Coefficient	Standard error	t statistic	p value
Untreated (CHIS 2021)	7.1	7.9	8.6	0.8	0.7
Partially treated (CHIS 2022)	7.6	8.5	9.0	0.9	0.6
Fully treated (CHIS 2022)	7.4	8.4	8.8	1.0	0.4
Difference from untreated (CHIS 2021)
Partially treated (CHIS 2022)	0.5	0.6	0.5	0.1	−0.2	−0.001	0.0009	−1.0	.364
Fully treated (CHIS 2022)	0.4	0.6	0.3	0.2	−0.3	−0.002	0.0007	−2.9	.009

Note.—Details may not sum to totals due to rounding. Completion rates are base-weighted by the address's inverse selection probability and limited to addresses with one appended phone number. The full set of regression coefficients is provided in table B.1 of the supplementary data online.

a

Partially treated CHIS 2022 waves are those that began outbound dialing before the implementation of the adaptive design (May 24, 2022). Fully treated CHIS 2022 waves are those that began outbound dialing on or after this date.

b

Regression results are from a linear regression of the cumulative completion rate on (i) fixed effects for individual CHIS waves, (ii) fixed effects for individual dialing categories, and (iii) difference-in-difference interactions. The coefficient shown is the coefficient on the difference-in-difference interaction between the indicator for the specified treatment group and the indicator for the posttreatment dialing category (4+ calls). Regression was estimated on a panel of 47 waves × 3 dialing categories. Each wave was weighted by its fielded sample size and standard errors were clustered at the wave level.

3.1.2 Wave-cohort approach

For each treatment group, figure 3 compares completion rates (conditional on at least three calls) between the high-RP and low-RP cohorts. In CHIS 2021, the low-RP cohort shows a lower completion rate, as expected. This difference is larger in CHIS 2022, which, under the parallel trends assumption, is attributed to the adaptive design. As shown in table 3, in the fully treated CHIS 2022 waves, the descriptive wave-cohort estimate (5) is a 0.2 percentage-point reduction in the conditional completion rate from the adaptive design, similar to the point estimate from the wave-dial method. The coefficient $\delta$ on the DiD interaction term from the wave-cohort regression [${\delta }_{\mathrm{partial}}$ and ${\delta }_{\mathrm{full}}$ from (6)] is again statistically significant for the fully treated CHIS 2022 waves.

Figure 3.

Adult Interview Completion Rate by Intervention Cohort and Treatment Group. Completion rates are base-weighted by the address’s inverse selection probability and limited to addresses with one appended phone number and at least three calls. Denominator is the fielded sample within each cell (84,593 for high-RP, 2021 untreated; 39,004 for low-RP, 2021 untreated; 21,585 for high-RP, 2022 partially treated; 10,397 for low-RP, 2022 partially treated; 53,668 for high-RP, 2022 fully treated; and 23,401 for low-RP, 2022 fully treated).

Table 3.

Effect of Adaptive Design on CHIS Completion Rate (among Cases with at Least Three Calls), Wave-Cohort Method

Treatment groupa	Completion rate, by adaptive design intervention cohortb (%)			Difference-in-difference regressionc
	Continue dialing (higher response propensity)	Stop dialing (lower response propensity)	Difference	Coefficient	Standard error	t statistic	p value
Untreated (CHIS 2021)	1.3	0.9	−0.4
Partially treated (CHIS 2022)	1.4	0.7	−0.6
Fully treated (CHIS 2022)	1.2	0.5	−0.6
Difference from untreated (CHIS 2021)
Partially treated (CHIS 2022)	0.0	−0.2	−0.2	−0.278	0.2318	−1.2	0.230
Fully treated (CHIS 2022)	−0.2	−0.4	−0.2	−0.403	0.1715	−2.3	0.019

Note.—Details may not sum to totals due to rounding. Completion rates are base-weighted by the address's inverse selection probability and limited to addresses with one available phone number and at least three calls. The full set of regression coefficients is provided in table B.2 of the supplementary data online.

a

Partially treated CHIS 2022 waves are those that began outbound dialing before the implementation of the adaptive design (May 24, 2022). Fully treated CHIS 2022 waves are those that began outbound dialing on or after this date.

b

Completion rates are among cases with at least three calls. For the untreated (CHIS 2021) waves, the intervention cohort indicates the treatment that the case would have received, based on its modeled response propensity, under the adaptive design rules used for CHIS 2022.

c

Regression results are from a logistic regression of individual completion status on (i) fixed effects for individual CHIS waves, (ii) fixed effects for the intervention cohort, and (iii) difference-in-difference interactions. The coefficient shown is the coefficient on the difference-in-difference interaction between the indicator for the specified treatment group and the indicator for the “stop dialing” cohort. The regression was limited to sampled cases with at least three calls (n = 232,648).

3.2 Demographic Composition of Responding Sample

While we have no specific hypothesis for the impact of the adaptive design on the demographic composition of the responding sample, the rationale for varying the RP cutoff threshold between modeled strata was to limit any effect on demographic representativeness. Figure 4 illustrates the wave-dial DiD results for the composition of the responding sample, showing the weighted prevalence of target demographics among cumulative sets of adult respondents as of no calls, up to three calls, and more than three calls. Again, under the parallel trends assumption, any effect of the adaptive design would be reflected in a change in the incremental effect of the last group of calls. However, for some demographics, the incremental effect of the first three calls (unaffected by the adaptive design) differs between CHIS 2021 and 2022, suggesting that the parallel trends assumption is less plausible for these outcomes.

Figure 4.

Cumulative Prevalence of Key Subgroups among Respondents by Number of Calls and Treatment Group. Percentages are base-weighted by the address’s inverse selection probability and limited to addresses with one appended phone number. Percentages are cumulative as of the specified number of calls; therefore, the denominator of each percentage is the cumulative number of respondents who received up to the specified number of calls. For 2021 untreated: n = 9,702 for no calls, 10,815 for up to 3 calls, and 11,871 for up to 4+ calls. For 2022 partially treated: n = 2,550 for no calls, 2,863 for up to 3 calls, and 3,052 for up to 4+ calls. For 2022 fully treated: n = 5,919 for no calls, 6,704 for up to 3 calls, and 7,043 for up to 4+ calls.

As shown in table 4, the wave-dial regressions (4) find no statistically significant effect of the adaptive design on the representation of these subgroups among CHIS 2022 respondents with two exceptions. First, in the partially treated CHIS 2022 waves, there was a significant increase of about 0.3 percentage points in the prevalence of African American adults. Second, the partially treated CHIS 2022 waves show a significant decrease of about 0.2 percentage points in the prevalence of Asian-language responses.

Table 4.

Effect of Adaptive Design on Prevalence of Key Subgroups among CHIS Respondents, Wave-Dial Method

Subgroup	Treatment groupa	Cumulative prevalence among respondents, by calling category (%)			Incremental increase in prevalence, by calling category (%)		Difference-in-difference regressionb
		No calls	1–3 calls	4+ calls	1–3 calls	4+ calls	Coefficient	Standard error	t statistic	p value
Hispanic	Untreated (CHIS 2021)	16.6	16.7	17.1	0.1	0.3
	Partially treated (CHIS 2022)	14.9	15.4	15.4	0.4	0.0	−0.179	0.2554	−0.7	.484
	Fully treated (CHIS 2022)	14.9	15.1	15.3	0.2	0.2	0.063	0.2225	0.3	.777
African American	Untreated (CHIS 2021)	4.6	4.9	5.0	0.3	0.2
	Partially treated (CHIS 2022)	4.4	4.4	4.9	0.0	0.5	0.826	0.3452	2.4	.017
	Fully treated (CHIS 2022)	4.4	5.0	5.3	0.6	0.3	0.482	0.2996	1.6	.108
Age 18–29	Untreated (CHIS 2021)	8.7	8.5	8.4	−0.3	0.0
	Partially treated (CHIS 2022)	5.6	5.5	5.5	−0.2	0.1	0.185	0.3942	0.5	.640
	Fully treated (CHIS 2022)	5.4	5.2	5.1	−0.2	0.0	−0.152	0.4031	−0.4	.706
Age 65+	Untreated (CHIS 2021)	34.3	35.4	35.0	1.1	−0.4
	Partially treated (CHIS 2022)	41.4	43.2	42.8	1.8	−0.4	−0.047	0.2541	−0.2	.855
	Fully treated (CHIS 2022)	38.6	40.0	40.1	1.4	0.1	0.307	0.2023	1.5	.129
Spanish response	Untreated (CHIS 2021)	2.2	2.4	2.6	0.1	0.2
	Partially treated (CHIS 2022)	1.2	1.5	1.7	0.3	0.2	0.397	0.4207	0.9	.346
	Fully treated (CHIS 2022)	1.4	1.6	1.7	0.2	0.1	0.318	0.3009	1.1	.291
Asian language response	Untreated (CHIS 2021)	1.7	1.9	2.0	0.1	0.2
	Partially treated (CHIS 2022)	2.1	2.2	2.1	0.0	0.0	−1.164	0.5325	−2.2	.029
	Fully treated (CHIS 2022)	1.9	1.9	2.1	0.0	0.2	0.295	0.3870	0.8	.445
No high school diploma	Untreated (CHIS 2021)	2.1	2.4	2.6	0.2	0.2
	Partially treated (CHIS 2022)	2.5	2.7	2.8	0.1	0.2	−0.022	0.5560	0.0	.968
	Fully treated (CHIS 2022)	1.5	2.0	2.0	0.5	0.0	−0.274	0.4149	−0.7	.509
Below 200% federal poverty line	Untreated (CHIS 2021)	13.3	13.8	14.3	0.5	0.4
	Partially treated (CHIS 2022)	12.8	13.4	13.6	0.6	0.2	−0.107	0.3310	−0.3	.746
	Fully treated (CHIS 2022)	11.0	11.9	12.1	0.9	0.2	−0.080	0.2322	−0.3	.730
Household with children	Untreated (CHIS 2021)	13.8	13.5	14.0	−0.3	0.6
	Partially treated (CHIS 2022)	9.7	9.2	9.5	−0.5	0.3	−0.062	0.2879	−0.2	.829
	Fully treated (CHIS 2022)	10.0	9.8	9.9	−0.2	0.1	−0.272	0.2613	−1.0	.298

Note.—Details may not sum to totals due to rounding. Percentages are base-weighted by the address's inverse selection probability and limited to respondents with one appended phone number. The full set of regression coefficients is provided in table B.3 of the supplementary data online.

a

Partially treated CHIS 2022 waves are those that began outbound dialing before the implementation of the adaptive design (May 24, 2022). Fully treated CHIS 2022 waves are those that began outbound dialing on or after this date.

b

Regression results are from a logistic regression of an indicator for the specified subgroup on (i) fixed effects for individual CHIS waves, (ii) fixed effects for individual dialing categories, and (iii) difference-in-difference interactions. The coefficient shown is the coefficient on the difference-in-difference interaction between the indicator for the specified treatment group and the indicator for the posttreatment dialing category (4+ calls). The regression was estimated on n = 21,966 respondents with one appended phone number.

For both groups, however, the preadaptive design trend (the impact of the first three calls) differed between CHIS 2021 and 2022 (figure 4). Furthermore, if the effect captured by the regression were wholly attributable to the adaptive design, we would expect it to be larger in the fully treated waves, which in fact show no significant effect. Finally, both effects disappear with an alternative, continuous coding of treatment exposure (see the supplementary data online). Taken together, these patterns cannot rule out, but do cast doubt on, the attributability of the change to the adaptive design.

As shown in the supplementary data online, the wave-cohort method also does not show a significant effect of the adaptive design on the prevalence of most of the target subgroups. However, statistical power for the wave-cohort method is limited by the small number of respondents from the low-RP cohort (63 in the partially treated waves and 127 in the fully treated waves).

3.3 Calling Effort

The purpose of the adaptive design was to control data collection costs by limiting the dialing and interviewing effort associated with the CATI operation. Therefore, we use the mean number of calls per sample unit (MCPS) as a proxy for the cost reduction (for a given sample size) obtained from the adaptive design. Table 5 shows a wave-cohort decomposition of the MCPS. As expected, the adaptive design decreased calling within the low-RP cohort, reducing the MCPS within by nearly one-third (relative to CHIS 2021) in the fully treated CHIS 2022 waves. Note that the MCPS remains above 3 even in the fully treated waves; this is because the call counts include inbound calls and, for operational reasons discussed in the supplementary data online, some cases were stopped at more than three calls. However, the effect was small at the overall treatment group level: overall, the MCPS was only about 4 percent lower in the fully treated CHIS 2022 waves than in CHIS 2021. This discrepancy is explained by the fact that, as shown in table 5, the MCPS increased for the high-RP cohort in CHIS 2022—that is, those cases not subjected to the adaptive design were dialed more intensively in CHIS 2022.

Table 5.

Mean Number of Calls per Sampled Address, by Treatment Group and Intervention Cohort

Treatment groupa	Intervention cohortb	Number of sampled addresses	Mean calls per sampled address		Percent of sampled addresses within treatment group
			By treatment group and intervention cohort	Overall by treatment group
Untreated (CHIS 2021)	Fewer than 3 calls	33,724	0.86	4.74	23.0
	3+ calls, continue dialing	84,593	5.90		48.2
	3+ calls, stop dialing	39,004	5.89		28.8
Partially treated (CHIS 2022)	Fewer than 3 calls	9,563	0.84	4.56	25.4
	3+ calls, continue dialing	21,585	6.10		44.6
	3+ calls, stop dialing	10,397	5.41		30.0
Fully treated (CHIS 2022)	Fewer than 3 calls	16,413	0.76	4.54	19.2
	3+ calls, continue dialing	53,668	6.35		49.7
	3+ calls, stop dialing	23,401	4.00		31.1

Note.—Details may not sum to totals due to rounding. Means and percentages are base-weighted by the address's inverse selection probability and limited to addresses with one appended phone number.

a

Partially treated CHIS 2022 waves are those that began outbound dialing before the implementation of the adaptive design (May 24, 2022). Fully treated CHIS 2022 waves are those that began outbound dialing on or after this date.

b

For the untreated (CHIS 2021) waves, the intervention cohort indicates the treatment that the case would have received, based on the number of calls and modeled response propensity, under the adaptive design rules used for CHIS 2022.

Thus, to estimate the reduction in dialing effort associated with the adaptive design, we estimate a counterfactual MCPS for the CHIS 2022 treatment groups. We assume that without the adaptive design, the MCPS in the low-RP cohort would have shown the same relative change from CHIS 2021 as the MCPS in the high-RP cohort. For example, in the high-RP cohort, the MCPS increased by about 7.6 percent (from 5.90 to 6.35) from CHIS 2021 to the fully treated CHIS 2022 waves; we therefore assume that without the adaptive design, the MCPS in the low-RP cohort would also have increased by 7.6 percent (from 5.89 to 6.34). We then calculate the counterfactual treatment-level MCPS by averaging the cohort-level MCPS, weighting each cohort by its share of the treatment group. This decomposition suggests that in the fully treated waves, the adaptive design reduced dialing effort, as measured by the MCPS, by about 14 percent relative to a modeled no-adaptive-design counterfactual (table 6).

Table 6.

Counterfactual versus Actual Mean Calls per Sampled Address, by Treatment Group

Treatment groupa	Mean calls per sampled address
	Counterfactualb	Actual	Difference
Untreated (CHIS 2021)	4.74	4.74	0.00
Partially treated (CHIS 2022)	4.76	4.56	−0.21
Fully treated (CHIS 2022)	5.27	4.54	−0.73

Note.—Details may not sum to totals due to rounding. Means are base-weighted by the address's inverse selection probability and limited to addresses with one appended phone number.

a

Partially treated CHIS 2022 waves are those that began outbound dialing before the implementation of the adaptive design (May 24, 2022). Fully treated CHIS 2022 waves are those that began outbound dialing on or after this date.

b

The counterfactual estimates the CHIS 2022 mean calls in the absence of the adaptive design by assuming that the mean calls in the “Stop dialing” cohort would have shown the same relative change from CHIS 2021 as the mean calls in the “Continue dialing” cohort.

4. DISCUSSION

The CHIS 2022 adaptive design aimed to reduce dialing while avoiding a meaningful reduction in the completion rate or change in the demographic composition of the responding sample. Based on a DiD comparison with CHIS 2021, these goals were largely met. In the fully treated CHIS 2022 waves, the mean calls per address (among addresses with an appended phone number) decreased by about 14 percent relative to a no-adaptive-design counterfactual, while the completion rate decreased by only about 3 percent in relative terms. We do not find strong evidence of demographic differences attributable to the adaptive design.

The CHIS 2022 adaptive design therefore provides evidence that RP modeling, applied to paradata from early calls, can reduce the variable cost of outbound dialing as a secondary contact mode in ABS studies. By using models to identify and stop cases for which further dialing is unlikely to be productive, the impact on completion rates and sample composition can be minimized. We stress that this dynamic approach required near-continual monitoring of the CATI operation, since not all cases reached the intervention point at the same time. The process of stopping cases, assigning RP scores, and retiring or re-starting cases was largely automated; however, set-up, testing, and monitoring entailed some overhead effort, which is an important consideration in any dynamic adaptive design. Given the size and complexity of the CHIS data collection, and the length of the field period, this fixed set-up cost was relatively trivial; we estimate that labor hours associated with the set-up and monitoring of the adaptive design comprised well below 1 percent of the total labor hours used for CHIS 2022. This set-up was also largely a one-time cost, and therefore its significance diminishes when cost savings from reduced dialing in future CHIS administrations (which continued to use the adaptive design) are considered. Altogether, the benefit-cost ratio to a similar adaptive design is more likely to be positive (i) the larger the preadaptive design dialing volume and/or (ii) for recurring collections in which the fixed set-up costs are effectively spread across multiple administrations. As a potential limitation, our estimated variable cost reduction rests on the assumption that, without the adaptive design, the mean calls per address would have shown a similar change in the low-RP as in the high-RP cohort. While we consider this assumption to be plausible—given that these cohorts would not have been operationally differentiated under the nonadaptive dialing protocol—we cannot fully account for ad hoc decision-making about dialing priorities that may have been made in a no-adaptive-design scenario.

We also stress that the stratification models for the presence of target subgroups were equally critical to the adaptive design as the RP model. A likely contributor to the muted impact on sample composition was the ability to allow more lenient RP thresholds in strata in which the CHIS 2021 data suggested that a reduction in dialing would have a larger effect. When control of sample composition is a factor in the optimization criteria for an adaptive design, this needs to be accounted for in the modeling strategy. There is an important distinction between targeting based on RP and targeting based on “bias propensity” (the likelihood that a case will contribute to nonresponse bias if it fails to respond) (Peytchev et al. 2022). In this adaptive design, the stratification models effectively functioned as bias propensity models. The recognition of this distinction, and the need to consider both in the intervention criteria, becomes particularly important when the intervention involves reducing effort for low-RP cases.

One limitation of the CHIS 2022 adaptive design was that the RP model was only moderately accurate (see the supplementary data online). A more accurate RP model would have lessened the tradeoff between dialing effort and completion rates, allowing dialing to be abbreviated for more cases without a further decrease in completion rates. As noted above, the predictors were limited to paradata from the first three calls and a handful of static predictors; the model omitted the appended commercial variables used for the stratification models. Their exclusion was motivated by the need to limit the impact of the adaptive design on respondent composition—recognizing that many of the target subgroups have historically shown lower-than-average response rates to CHIS, we aimed to avoid mechanically allocating higher shares of target demographics to the lower-RP cohorts for which dialing ended early. However, if these additional predictors had meaningfully improved model accuracy, it is possible that any negative effect of their inclusion in the RP model would have been offset by an improved ability to identify units for which additional dialing would not be productive. Evaluation of the tradeoffs of including substantive predictors in RP models—again, particularly when effort is reduced for low-RP cases—is an important direction for additional research.

Finally, because of the nonexperimental implementation, these conclusions rely on a DiD analysis and therefore on the crucial parallel trends assumption. On the one hand, these results demonstrate how quasi-experimental methods can be useful for survey methodologists when randomized experimentation is not feasible—such as when a design change must be applied to an entire sample for budgetary reasons. On the other, when considering a nonexperimental design change, it is crucial to consider the underlying assumptions of any planned quasi-experimental evaluation, and whether these are plausible in the context of the particular collection. In the case of CHIS 2022, we considered the DiD analysis to be plausible a priori because a sample from the immediate prior year, with a similar design except for the adaptive design, was available as a nontreated sample. In general, the DiD approach is likely to be most viable in similar situations. Even so, we found that for some outcomes of interest—specifically, some demographic indicators—preadaptive-design trends suggest that the parallel trends assumption may not have held. For those outcomes, the results reported here must be taken with caution. Other quasi-experimental strategies—such as regression discontinuity methods (e.g., Peytchev et al. 2022)—can be considered in contexts in which the parallel trends assumption cannot plausibly be met.

Supplementary Materials

Supplementary materials are available online at academic.oup.com/jssam.

Funding support for the research described in this article was provided by the University of California, Los Angeles (UCLA) Center for Health Policy Research.

This analysis uses nonpublic CHIS sample and paradata files, which are used with the permission of the sponsor. Views expressed in this article are those of the authors and not the study sponsor nor any organization with which the authors are affiliated. This study design and analysis was not preregistered. Analytic software code is available upon request.