Usefulness of Internet Surveys to Identify People with Disabilities: A Cautionary Tale

Abstract

Disability is an important characteristic to consider in survey research. However, people with disabilities are a hard-to-reach population. Internet survey methods offer tremendous potential to expand researchers’ ability to reach and learn about people with disabilities. The goal of this study is to examine potential bias when using nonprobability Internet samples to investigate demographics and socioeconomic outcomes of people with disabilities. We compare the findings based on a national employment and disability survey instrument fielded to four samples: (1) a random-digit dial (RDD) sample, (2) a prescreened sample from a nonprobability Internet access panel, for which screening was based on the presence of 139 previously reported health conditions, (3) an unscreened sample from another nonprobability Internet access panel (without previously prescreened health conditions), and (4) a mixed nonprobability self-recruited (river and snowball) sample. Each sample was weighted on four demographic variables (gender, age, race/ethnicity, and region) using benchmarks from the American Community Survey (ACS). Three dichotomous outcome variables of interest (level of education, household income, and current employment status) were contrasted with weighted population estimates from the ACS. Results showed that the sample resulting from the RDD and all three nonprobability Internet samples differed significantly from ACS population estimates on all three outcome variables. Reweighting to include type of functional disability did not significantly reduce dissimilarities with ACS for any of the four samples. Nonprobability Internet survey methods offer relatively low-cost, easy-to-use avenues for disability-related research. Yet, researchers must proceed with caution to reduce or avoid known sources of bias in both the methodology and the interpretation of results.

1. INTRODUCTION

People with disabilities represent a substantial portion of the US population, with estimates as high as 23.7 percent of US noninstitutional civilians of eighteen to sixty-four years old in 2014 (Taylor 2014). From a social policy perspective, people with disabilities represent substantial portions of the people who are not employed, living in poverty, participating in government safety net programs, and receiving poor-quality health care. From a marketing perspective, households with at least one person with a disability represent approximately 18 percent of spending in the United States (Imparato et al. 2010) . In the context of survey methods, “entities involved in survey data collection—such as federal agencies, university survey centers, and private polling firms—should (and sometimes by law must) consider the extent to which their methods create barriers to survey participation for people with disabilities” (Markesich, Feldman, and Rafferty 2019, p. 1).

However, disability is a complex and dynamic phenomenon and is difficult to operationalize in surveys. Disabilities vary greatly by a host of key characteristics, such as type (e.g., physical, mental, and sensory), underlying health condition (e.g., injury, illness, and genetics), severity, onset date, mitigating factors (e.g., use of assistive devices), environmental context (e.g., inaccessible buildings), and possible multiple disabilities. The American Community Survey (ACS), Current Population Survey (CPS), and Behavioral Risk Factor Surveillance System (BRFSS) use six questions to identify the population with disabilities. In contrast, the 2014 Survey of Income and Program Participation (SIPP), Social Security Administration Supplement (formerly several topical modules)—perhaps the most comprehensive disability-related survey in the United States—used 142 questions and relatively complex skip patterns to capture several key characteristics, such as severity, underlying health condition, and onset date of the main underlying condition.

Internet survey methods offer tremendous potential to expand the ability of survey researchers to reach and learn about people with disabilities. Like computer-assisted interviews, Internet questionnaires with complex skip patterns allow for inquiry about details that may vary by disability type. Like mail questionnaires, Internet questionnaires provide privacy and thus may reduce social desirability bias when asking about the need for assistance. Perhaps most advantageous is the potential use of Internet access panels to readily obtain samples of people with specific and potentially rare (i.e., low incidence) underlying health conditions.

However, caution is needed when using Internet survey methods, particularly nonprobability Internet access panels from uncertain sampling frames (Callegaro, Villar, Yeager, and Krosnick 2014). Internet access panels may be created to support market research and not population studies. The AAPOR Task Force recommends that researchers should “avoid nonprobability online panels when one of the research objectives is to accurately estimate population values.” (Baker et al. 2010, p. 5). Nonprobability self-recruited samples (a.k.a., river samples) based on responses to website banner ads should likely be used with similar caution.

The goal of this article is to examine potential bias when using nonprobability Internet samples to investigate demographics and socioeconomic outcomes of people with disabilities. We compare the findings based on a national employment and disability survey instrument fielded to four samples: (1) a random-digit dial (RDD) sample, (2) a prescreened sample from a nonprobability Internet access panel, for which screening was based on the presence of 139 previously reported health conditions, (3) an unscreened sample from another nonprobability Internet access panel (without previously prescreened health conditions), and (4) a mixed nonprobability (river and snowball) sample derived from invitations to known email addresses (with prompting to share the invitation with others) and self-recruiting to links on two web pages. Key items in the survey were based on the items in the ACS, enabling us to contrast our findings with findings from the ACS, which provides benchmarks for the intended target population.

1.1 Use of Internet Surveys for Research

Web-based applications and software have dramatically eased the use of Internet surveys for research with many different population-based and nonpopulation-based samples (Wright 2005). This ease of use occurs along several lines. First, with regard to survey design and programming, applications from companies such as Survey Monkey and Qualtrics simplify point-and-click survey construction and offer templates for dozens of question types as well as entire libraries of sample questions. Many, although not all, question types can be mobile-optimized and thus suit respondents utilizing smart phones to answer surveys (Ansolabehere and Schaffner 2014).

Second, the responses of Internet survey participants also appear to be less likely to be influenced by social desirability (Baker et al. 2010; de Leeuw 2012; Stern, Bilgen, and Dillman 2014). Surveys of people with disabilities may benefit from the privacy afforded by the use (or optional use) of online and mail questionnaires because information about the need for disability-related supports and services do not need to be divulged to interviewers and thus may lead to more truthful responses. Moreover, many participants strongly prefer self-administered surveys (Ansolabehere and Schaffner 2014).

Third, for researchers, Internet surveys are typically much more affordable and expedient (i.e., rapid access to samples) than other survey approaches, such as in-person or telephone interview (Couper 2007; Ferraro, Krenzke, and Montaquila 2008; Ansolabehere and Schaffner 2014). Web-based application and software companies provide ready access to samples derived from “access panels” of persons recruited to participate in future online surveys. Invitations to participate in a specific survey may be sent to a subsample of panel members based on their demographic characteristics.

APPOR (2016) describes four common types of Internet surveys based on the combinations of sample methods (probability or nonprobability) and sampling frames (known or unknown frames):

• Specifically Named Persons Sample—a sample of specifically named persons probabilistically drawn from a list-based sampling frame of target populations, such as a list of the email addresses of the target population,

• A Sample of Probability Internet Panel Members—a sample of persons drawn from an “access” panel of persons probabilistically recruited from a known sampling frame of the target population (e.g., recruited via an RDD sampling frame) to participate in future online surveys, where recruitment may be broadened by providing Internet access to sample members when needed,¹

• A Sample of Nonprobability Internet Panel Members—a sample of persons drawn from an “access” panel of persons nonprobabilistically recruited from an unknown sampling frame (e.g., persons who “opt-in” to a panel by accepting an invitation offered to the unknown sampling frame of persons visiting a website) to participate in future online surveys, and

• Self-Recruited Online Sample—a sample of persons nonprobabilistically “self-recruited” to participate in a survey(s) by clicking on banner/pop-up ads, from some unknown sampling frame of persons exposed to the ad (a.k.a., river sample).

In recent years, declining response rates and increasing costs associated with RDD surveys have become barriers to successful research using this method (Dillman and Smyth 2007; Ferraro et al. 2008). At the same time, proliferation of Internet survey technology makes web-based options increasingly attractive, especially with low incidence populations (Riggle, Rostosky, and Reedy 2005; Wright 2005; Couper 2007). Investigating populations with specific low-incidence disability types has been a Federal strategic research priority (National Institute on Disability and Rehabilitation Research [NIDRR] 2013).

1.2 Use of Internet Surveys for Research with People with Disabilities

The initial step in considering disability in survey research most likely involves determining how disability will be defined and subsequently operationalized. Disability may be conceptualized in many ways, since it is not just the presence of a health condition but a dynamic interaction between the person and the context within which the person functions (World Health Organization 2019). It concerns whether a health condition or environmental context limits a person’s ability to perform a human function (e.g., seeing), task (e.g., reading), activity (e.g., fill out a paper survey), and/or social role (e.g., vote), with the potential of mitigating and exacerbating factors (e.g., large print materials and lack of appropriate Braille signage). In Federal code, there are more than sixty-seven different definitions of disability (New Editions 2018).

In an effort to address health disparities, Section 4302 of the Patient Protection and Affordable Care Act (ACA) of 2010 mandates the establishment of standards for the collection and dissemination of health statistics of specific demographic subpopulations, including people with disabilities. In October 2011, the US Department of Health and Human Services (USDHHS) selected a six-question sequence, which was developed for and first introduced into the ACS in 2008, to be the minimum standard for the set of questions to identify people with disabilities in all national federal data collection efforts (USDHHS 2011). This sequence appeared in the 2019 ACS questionnaire as follows:

• E. Answer questions about PERSON 1 on the next page if you listed at least one person on page 2. Otherwise, SKIP to page 28 for the mailing instructions.

• 16a. Is this person deaf or does he/she have serious difficulty hearing? [Yes | No]

• b. Is this person blind or does he/she have serious difficulty seeing even when wearing glasses? [Yes | No]

• F. Answer questions 17a–c if this person is five years old or over. Otherwise, SKIP to the questions for Person 2 on page 12.

• 17a. Because of a physical, mental, or emotional condition, does this person have serious difficulty concentrating, remembering, or making decisions? [Yes | No]

• b. Does this person have serious difficulty walking or climbing stairs? [Yes | No]

• c. Does this person have difficulty dressing or bathing? [Yes | No]

• G. Answer question 18 if this person is fifteen years old or over. Otherwise, SKIP to the questions for Person 2 on page 12.

• 18. Because of a physical, mental, or emotional condition, does this person have difficulty doing errands alone such as visiting a doctor’s office or shopping? [Yes | No]

Based in part on these questions, an ever-growing body of evidence demonstrates that people with disabilities in the United States experience health disparities and are under-represented in the labor force (Brucker and Houtenville 2015). Despite striving to work, people with disabilities face many barriers to employment (Sundar et al. 2018), including lower wages and less desirable positions (Schur, Kruse, Blasi, and Blanck 2009; Brucker and Houtenville 2015). Better understanding factors associated with these inequities helps ensure that policies and interventions aimed at improving outcomes for this population enjoy maximum efficiency and effectiveness.

However, people with disabilities are a hard-to-reach population. Gathering sufficiently large or representative samples of people with disabilities remains a significant challenge, in part because of the heterogenous, temporal, and contextual nature of disability. Survey research with people with disabilities sometimes requires oversampling to ensure minimum acceptable representation of this population. As such, the already expensive cost of telephone surveys can quickly surpass available budgets. Internet-based samples present a potentially attractive alternative, with lower marginal costs per complete response (Berrens, Bohara, Jenkins-Smith, Silva, and D. L. Weimer 2003). In this study, the marginal costs per complete response of the Internet surveys ranged from $0 to $15, whereas it was $105 for the RDD survey. Yet, the use of Internet-based panels for research among people with disabilities may exacerbate coverage bias.

Recent estimates of Internet access suggest that between 11 and 30 percent of adults in the United States do not regularly use the Internet (Baker et al. 2010). People with disabilities are less likely to have cell phones (Morris, Jones, and Sweatman 2016), and they are more likely to have lower incomes and less education than people without disabilities (Brucker and Houtenville 2015). As these factors correlate with Internet access, it seems likely that people with disabilities would participate in online-panels less frequently than people without. Nevertheless, online opt-in panels may offer targeted panels (subgroups) clustered by specific characteristics or sample routing algorithms (Couper 2007; Baker et al. 2010) that may facilitate reaching people with disabilities in a faster and more efficient manner. For example, part of the research described in this article used a sample drawn from a panel of individuals who had reported at least one of the 139 ailments or health conditions. Even when targeted panels are not available, oversampling with Internet surveys is dramatically more expedient and less costly than telephone interviews. Another benefit of online surveys concerns the lower likelihood of responding in such a way as to conform with social desirability (Baker et al. 2010; de Leeuw 2012; Stern et al. 2014).

Although ease of construction, reduced social desirability bias, participant preference, access to sample, and lower cost are compelling, they are not sufficient to ensure robust research findings. Even if social desirability constitutes less of a concern, other factors influencing self-report data from online sources, such as satisficing, have been documented (Chang and Krosnick 2010; Hays, Liu, and Kapteyn 2015). To maximize reliability and validity of findings, investigators interested in the benefits and availability of web-based research with people with disabilities need to understand how results might differ based on different survey modes. Subsequently, the purpose of our research is to examine the usefulness of four different samples—one RDD, two nonprobability Internet access panels, and a self-recruited river and snowball sample—to estimate outcomes of interest among people with disabilities.

2. ANALYSIS OF DATA

2.1 Data Sources

Table 1 shows a summary of survey characteristics from the four different samples that were collected and compared. Each is described in detail next.

Table 1.

Summary of Survey Characteristics

	RDD	Health Cond Panel	Gen Pop Panel	Self-Recruited Web
Period of data collection	10/17/2014–4/23/2015	5/10/2016–6/1/2016	3/29/2019–4/10/2019	4/28/2015–10/3/2015
Duration of data collection (days)	189	22	13	159
Mode	Telephone	Web browser	Web browser	Web browser
Sample frame	Landline/cell listings less nonresidential listings	Unknown frame, prescreened for health condition(s)	Unknown	National and regional listservs and social media postings
Incentives	None	Unknown value	Unknown value	None
Proxy allowed	Yes	No	No	Yes
Contacted	117,871	Unknown	Unknown	Unknown
Accessed	Not applicable	11,045	6,350	682
Ineligible	95,347*	7,440	2,665	Not ascertained
Incomplete	12	0	0	65
Screened out for inattentiveness	Not applicable	583	664	Not applicable
Completed	3,013	3,022	3,021	579
Analytic sample, w/functional disability	2,126	1,912	969	430
Proxy responses	16%	Not applicable	Not applicable	34%
Time to complete (minutes)	Mean, 18	Median, 13	Median, 4	Median, 11
Marginal costs per complete	$105	$6.50	$9.50 or $15.00	$0.00

*

Estimated, applying a 14 percent prevalence rate to nonrespondents.

2.1.1 RDD telephone survey

The national employment and disability RDD survey, conducted by a university survey center, targeted households that included at least one person between eighteen and sixty-four years, who experienced a disability. A randomly selected sample of 117,871 telephone numbers were selected from an RDD dual sample frame of roughly 50 percent landline numbers and 50 percent cell phone numbers to reduce the potential noncoverage bias for households that use cell phones exclusively (Brick, Dipko, Presser, Tucker, and Yuan 2006). Business listings and other nonresidential listings were excluded from this sample frame. If more than one person in the household had a disability or health condition, the person with the last birthday in the calendar year was selected to respond. Proxy responses were allowed for those who could not complete the telephone interview.

Informed consent to participate was obtained in accordance with the university’s Institutional Review Board (IRB). This sample of 117,871 telephone numbers yielded 3,013 complete interviews, twelve partially completed interviews, and 3,977 noninterviews (e.g., refusals) from households deemed to contain an eligible participant. Among the other 110,869 telephone numbers, an estimated 15,522 interviews with eligible persons were possible, assuming 14 percent of households include at least one person between eighteen and sixty-four years with a disability. This suggests an AAPOR Response Rate 3 of 13.4 percent was achieved, that is, RR3 = 3,013/(3,013 + 12 + 3,977 + 15,522).

Interviews were conducted over a 189-day period between October 2014 and April 2015. The average complete interview length was eighteen minutes. The marginal cost per complete response was approximately $105.00.

Of the 3,013 interviews completed, 70.6 percent of participants indicated at least one functional disability (hearing, vision, ambulatory, or cognitive disability), resulting in an analytic sample 2,126 participants. About 16 percent of the analytic sample was through proxy respondents because the person with the disability was unable to complete the telephone interview independently.

2.1.2 Nonprobability Internet panel prescreened for health conditions.

An Internet version of the national employment and disability telephone interview instrument was created using the Qualtrics web-based application. It was offered to adult members of a nonprobability Internet panel maintained by Qualtrics and its partner organizations. Panel members were recruited from an unknown sampling frame(s) using a variety of methods, including web intercept, targeted email lists, panel member referral, and social media. Incentives for respondents included cash payments, free downloads, and/or membership points; all incentives were decided and allocated by Qualtrics and its partners. Informed consent to participate was obtained in accordance with the requirements of the IRB at the researchers’ institution, and participants were verified by Qualtrics through a double opt-in process.

Invitees to the survey included adults who were prescreened members of a health conditions panel, meaning that they had previously indicated to Qualtrics the presence of at least one of the 139 ailments or health conditions. Conditions ranged from mild conditions such as psoriasis or acne to more serious acute or chronic conditions such as AIDS, dementia, or stroke. General disability or specific functional disabilities was not included in the list, and the researchers had no knowledge about which of the listed ailments respondents had experienced.

Between May 10 and June 1, 2016 (a 22-day period), the survey was accessed by 11,045 panel members. Of those, 4,259 were screened out because they did not answer affirmatively to any of the study’s disability questions, and 3,181 were not admitted to the survey for being more than sixty-four years. Another 583 were dropped for inattentive responding, which means that participants incorrectly answered at least one Likert-type item designed to assess whether the questions were being thoroughly read. The median time to complete the survey was thirteen minutes. As there were several different tracks (i.e., skip patterns) through the survey, and some were very short, no participants were excluded based on time to complete the survey. Instead, responses with very short duration times were reviewed individually to verify that they belonged to the shortest survey track. This resulted in no further exclusions. The marginal costs per complete response were $9.50 or $15.00, depending on the order in which the questions appeared for the respondent.

Of the remaining 3,022 participants, 63.3 percent indicated one or more functional disabilities (hearing, vision, ambulatory, or cognitive disability) and were retained as the analytic sample for this study (n = 1,912). This survey did not include any proxy respondents.

2.1.2 Nonprobability Internet panel (general population, not prescreened).

To compare the prescreened health conditions panel with a general population panel (not prescreened for ailments or health conditions), another employment and disability survey was fielded using a Qualtrics nonprobability panel. Inclusion criteria for survey respondents again were working-age adults between eighteen and sixty-four years living in the United States. The median time to complete the survey was four minutes.

Between March 29 and April 10, 2019 (a thirteen-day period), the survey was accessed by 6,350 individuals; 5,444 provided informed consent in accordance with protocols of the university’s IRB. Of those, 664 screened out for fast (less than two minutes) or inattentive responding (i.e., they incorrectly responded to one Likert-type item designed to test they were paying attention). Another 1,759 potential respondents were terminated for being over quota; quotas were established for this survey to ensure half of the participants had a disability and half did not. Of the remaining 3,021 respondents, 49 percent had any disability and 32 percent had at least one functional disability (hearing, vision, ambulatory, or cognitive disability). The latter comprised the analytic sample for this study (n = 969). This survey did not include any proxy respondents. The marginal cost per complete response was approximately $6.50.

2.1.3 Nonprobability self-recruited river and snowball sample.

Early on in the effort of the original national employment and disability survey, people from the disability advocacy community wanted to learn how they could participate in the study. Rather than turn them away, we posted the Internet version of the instrument online. We sent invitations to take the survey via our own and partner listservs (both national and regional). Social media was also used to promote participation in the survey. Recruitment messages conveyed that the “employment” survey was open to adults with disabilities or their proxy responders. The survey was open between April 28 and October 3, 2015 (a 159-day period). A total of 682 people clicked on the survey link; of those, 579 provided informed consent and met the inclusion criteria (eighteen to sixty-four years with at least one disability and living in the United States). Among those who met the inclusion criteria, sixty-five did not answer any survey questions and were dropped. Of those, 84 percent had at least one functional disability (hearing, vision, ambulatory, or cognitive disability) and comprised the analytic sample for this study (n = 430). Proxy responders made up 34 percent of the sample; thirty-two individuals took the survey twice, answering once for themselves and again as a proxy for someone else. The marginal costs per complete response was essentially $0, free.

2.1.4 American Community Survey.

The ACS was used to estimate the characteristics of the US population for reference as benchmarks to which the estimates from the other four samples were compared. The ACS is the primary source of demographic and socioeconomic statistics in the United States. It is a nationally representative sample of approximately two million households and 150,000 group quarters, annually. Our sample is derived from the 2017 ACS five-year pooled public-use microdata file and contains 1,006,512 individuals of eighteen to sixty-four years living in households and noninstitutional group quarters.

2.2 Analytical Approach

The overall goal of the research was to compare the demographic composition and socioeconomic outcomes of people with disabilities across the four data sources, relative to the ACS. Optimally, such as study would compare disability prevalence rates (percentages of a sample that report disabilities) across the data sources before moving on to demographics and outcomes. However, disability prevalence rates could not be estimated with the RDD, health conditions panel, and self-recruited samples because people without disabilities were screened out of the surveys. The decision to exclude people without disabilities was based on the primary intent of the first study (the RDD study), which was to investigate issues specific to workers and job seekers with disabilities. The opportunity to compare the RDD sample to nonprobability Internet panel samples arose when additional funding became available, but not enough funding to afford a more expensive probability Internet panel sample.

2.2.1 Sample weights.

Weighted proportions from the ACS Integrated Public Use Microdata Series (IPUMS) 2017 five-year estimates (Ruggles et al. 2019) were used to generate poststratification weights based on the US adults of eighteen to sixty-four years who reported at least one functional disability (hearing, vision, ambulatory, or cognitive disability). These weights were based on four demographic variables: gender (male, female); age (eighteen to twenty-four, twenty-five to thirty-five, thirty-five to forty-four, forty-five to fifty, and fifty-five to sixty-four years); race/ethnicity (White only, non-Hispanic; Black only, non-Hispanic, other/multiple race, non-Hispanic; and Hispanic); and Census region (Northeast, Midwest, South, and West). We also used weights that added disability type (hearing only, vision only, ambulatory only, cognitive disability only, and multiple disabilities).

2.2.2 Analysis.

Difference-in-proportion, difference-in means, and Pearson chi-squared tests were conducted, using Stata IC/15 (College Station, TX) to assess differences between estimates from each of our survey samples and the corresponding ACS benchmark.

2.2.3 Disability variables.

Disability was identified in each of the four survey samples using the four functional disability questions from the six-question sequence in the ACS. Specifically, participants reported that (1) they had serious difficulty hearing; (2) they were blind or had serious difficulty seeing even when wearing glasses; (3) they had serious difficulty walking or climbing stairs; and/or (4) because of a physical, mental, or emotional condition, they had serious difficulty concentrating, remembering, or making decisions.

2.2.4 Outcome variables.

In addition to functional disability type, outcome variables for this study included education, income, and current employment status. The three socioeconomic variables were dummy-coded, and a value of one indicated college education or higher, mean annual household income of $30,000 or less, and employment status of currently employed. The disability variable included five mutually exclusive categories: hearing only, vision only, ambulation only, and cognitive only, and multiple disabilities (two or more of the previous types).

3. RESULTS AND DISCUSSION

Table 2 shows the unweighted proportions of the demographic variables, by sample. These demographic characteristics were used to calculate the initial sample weights. Pearson’s chi-squared tests were computed to compare the differences in the proportions of each collected sample to the ACS proportions. Region did not differ significantly from ACS in the RDD sample or in the general population nonprobability panel. All other unweighted proportions differed significantly from ACS estimates with significant Pearson’s chi-squared values ranging from χ² = 4.42 (p < .05) to χ² = 302.58 (p < .001).

Table 2.

Unweighted Proportions of Demographic Variables Among People With Disabilities, by Data Source

	ACS		RDD			Health Cond Panel			Gen Pop Panel			Self-Recruited Web
	%	SE	%	SE	χ2	%	SE	χ2	%	SE	χ2	%	SE	χ2
Gender
Male	51.7	0.0005	55.1	0.0108	9.81**	38.4	0.0111	135.06***	27.4	0.0143	228.81***	46.6	0.0243	4.42*
Female	48.3	0.0005	44.9	0.0108		61.6	0.0111		72.6	0.0143		53.4	0.0243
Age (years)
18–24	8.2	0.0003	6.0	0.0052	46.67***	1.8	0.0031	153.53***	15.7	0.0117	302.58***	17.1	0.0198	98.09***
25–34	11.6	0.0003	8.7	0.0062		10.1	0.0069		21.1	0.0131		22.0	0.0218
35–44	13.8	0.0003	13.5	0.0075		14.8	0.0081		22.7	0.0135		15.2	0.0188
45–54	25.9	0.0004	30.2	0.0101		22.8	0.0096		19.7	0.0128		22.6	0.0219
55–64	40.5	0.0005	41.7	0.0108		50.5	0.0114		20.9	0.0131		23.1	0.0221
Race/ethnicity
White only	66.6	0.0005	74.8	0.0096	75.35***	82.8	0.0087	227.68***	75.3	0.0139	39.33***	81.9	0.0203	62.73***
Black only	15.1	0.0003	12.9	0.0074		6.3	0.0056		9.1	0.0092		3.6	0.0098
Hispanic	6.5	0.0002	6.6	0.0055		6.9	0.0045		9.2	0.0079		5.0	0.0154
Other/multi	11.8	0.0003	5.8	0.0052		4.0	0.0058		6.4	0.0093		9.4	0.0115
Region
Northeast	16.1	0.0004	15.8	0.0080	3.52	19.4	0.0090	44.04***	16.7	0.0120	1.32	31.3	0.0247	69.71***
Midwest	22.2	0.0004	23.6	0.0093		25.2	0.0099		22.7	0.0135		25.0	0.0231
South	40.8	0.0005	39.2	0.0107		33.7	0.0108		41.3	0.0158		27.8	0.0239
West	20.9	0.0004	21.4	0.0090		21.7	0.0094		19.4	0.0127		15.9	0.0195

Notes.—ACS, American Community Survey; RDD, random-digit dial; Health Cond Panel, Qualtrics health conditions panel; Gen Pop Panel, Qualtrics general population panel; Self-Recruited Web, self and partner-recruited online convenience sample.

Pearson’s chi-squared (χ²) tests for differences in unweighted proportions of each sample to ACS proportions.

*

p < .05;

**

p < .01;

***

p < .001.

These finding are consistent with demographic patterns of Internet access with greater access tending to be among younger persons and persons from minority subpopulations (Baker et al. 2010). The unweighted result for region is consistent with the fact that the RDD study made a distinct effort to ensure that timing of the calls considered the US time-zones and both business and non-business hours. Researchers using Internet panel samples and response quotas that are easily filled may want to stagger when responses are solicited in batches to reflect regional patterns in Internet use.

Table 3 shows the weighted percentages, by data source, of the socioeconomic characteristics used as outcome variables in this study. t-Tests revealed significant differences in the weighted percentage of each collected sample compared to the weighted ACS on all three outcome variables. In the ACS, 34.3 percent of respondents were currently employed. In the three Internet samples, respondents were significantly more likely to report being currently employed. The self-recruited web sample, at nearly double the ACS percentage, showed the greatest difference (66.4 percent), followed by the general population panel (51.6 percent), and the health conditions panel (41.9 percent). The percentage of currently employed in RDD sample was closest to the ACS sample (36.6 percent).

Table 3.

Weighted Percentages of Socioeconomic Characteristics Among People With Disabilities, by Data Source

	ACS		RDD			Health Cond Panel			Gen Pop Panel			Self-Recruited Web
	%	SE	%	SE	t	%	SE	t	%	SE	t	%	SE	t
Currently employed	34.3	0.0005	36.6	0.0105	−2.08*	41.9	0.0113	−4.70***	51.6	0.0163	−8.20***	66.4	0.0228	−10.23***
Four-year college degree or higher	13.2	0.0005	29.1	0.0099	15.30***	41.1	0.0113	17.64***	25.3	0.0142	6.37***	47.2	0.0262	8.68***
Ann. household income < $30k	39.4	0.0003	44.3	0.0125	3.63***	34.2	0.0109	−3.48**	44.4	0.0162	2.34***	48.9	0.0274	2.32*

Notes.—Weighted by gender, age, race/ethnicity, and region.

t-Tests for differences in weighted mean of each sample to ACS mean.

*

p < .05;

**

p < .01;

***

p < .001.

In the ACS, 13.2 percent reported a four-year college degree or higher. All four collected samples were significantly more likely to report a four-year college degree or higher. The highest percentage was among the self-recruited web sample (47.2 percent) followed by the health conditions panel (41.1 percent), the RDD sample (29.1 percent), and the general population panel (25.3 percent).

In the ACS, 39.4 percent reported an annual household income of <$30,000. All four collected Internet samples were significantly more likely to report an annual household income of <$30,000. The highest percentage was again among the self-recruited web sample (48.9 percent) followed by the general population panel (44.4 percent), the RDD sample (44.3 percent), and the health conditions panel (34.2 percent).

These results for the Internet samples may reflect potential sample selection bias based on Internet access. Internet access is typically positively associated with educational attainment and income (Baker et al. 2010; Ryan 2018). Employment estimates may reflect that many people access the Internet via the workplace (Baker et al. 2010). The three Internet samples, with known sampling frames, produced employment estimates well above the ACS employment estimates, whereas the RDD sample, which uses a sampling frame of residential landline/cell phone listings, produced an employment estimate closest to the ACS estimate. To get a better understanding of this potential bias, researchers may want to ask respondents where they are accessing the Internet when they fill out the questionnaire, as well as stratify analyses by employment, income, and educational attainment.

Disability type is an important factor to consider when studying socioeconomic outcomes of people with disabilities (NIDRR 2013). Table 4 lists the unweighted proportions of disability type (in five mutually exclusive categories) for each data source. Pearson’s chi-squared tests showed that proportions of each collected sample differed significantly from the ACS proportions. Chi-squared values ranged from χ² = 417.36 (p < .001) for the RDD sample, to χ² = 246.75 (p < .001) for the health conditions panel, to χ² = 148.34 (p < .001) for the general population panel, to χ² = 25.42 (p < .001) for the self-recruited web sample. Compared to the ACS, the RDD sample and both Internet panel samples included substantially more respondents reporting multiple functional disabilities. Although 27.8 percent of ACS respondents had multiple functional disabilities, 45.1 percent in the RDD sample, 41.8 percent in the general population panel, and 37.7 percent in the health conditions panel reported the same. The self-recruited web sample was closest to the ACS for this group at 29.1 percent. However the self-recruited web sample resulted in a much higher percentage of respondents with cognitive disabilities (34.9 percent compared to 28.5 percent in the ACS) and much lower percentages of respondents with hearing only (5.4 percent compared to 11.2 percent in the ACS) or vision only disabilities (5.4 percent compared to 8.6 percent in the ACS). Regarding cognitive disabilities, respondents of both the RDD sample (14.2 percent, which did allow proxy responses, and the health conditions panel (13.7 percent), which did not allow proxy responses, appeared much less likely to report cognitive disabilities than ACS respondents (28.5 percent).

Table 4.

Unweighted Proportions of Mutually Exclusive Disability Type Among People With Disabilities, by Data Source

	ACS		RDD			Health Cond Panel			Gen Pop Panel			Self-Recruited Web
	%	SE	%	SE	χ2	%	SE	χ2	%	SE	χ2	%	SE	χ2
Hearing only	11.2	0.0003	14.0	0.0075	417.36***	13.2	0.0077	246.75***	9.3	0.0093	148.34***	5.4	0.0109	25.42***
Vision only	8.6	0.0003	7.8	0.0058		12.0	0.0074		12.5	0.0106		5.4	0.0109
Ambulatory only	23.9	0.0004	19.0	0.0085		23.5	0.0097		12.5	0.0106		25.4	0.0210
Cognitive only	28.5	0.0004	14.2	0.0076		13.7	0.0079		23.9	0.0137		34.9	0.0230
Multiple	27.8	0.0004	45.1	0.0108		37.7	0.0111		41.8	0.0158		29.1	0.0219

Pearson’s chi-squared (χ²) tests for differences in unweighted proportions of each sample to ACS proportions.

*

p < .05;

**

p < .01;

***

p < .001.

Table 5 lists the weighted proportions of disability type by data source. Pearson’s chi-squared values are substantially lower than those computed for the unweighted proportions, but each remains statistically significant. In order of magnitude, the weighted Pearson’s chi-squared for the RDD sample was χ² = 20.31 (p < .001), χ² = 11.86 (p < .001) for the health conditions panel, χ² = 9.30 (p < .001) for the general population panel, and χ² = 1.61 (p < .01) for the self-recruited web sample. It is important to note that the chi-squared tests have not received the Rao–Scott correction for the design effect; thus, these findings may appear to be somewhat more highly significant than they would otherwise be. For nonprobability panels, it is unclear what the design effect would be.

Table 5.

Weighted Proportions of Disability Type Among People With Disabilities, by Data Source

	ACS		RDD			Health Cond Panel			Gen Pop Panel			Self-Recruited Web
	%	SE	%	SE	χ2	%	SE	χ2	%	SE	χ2	%	SE	χ2
Hearing only	11.0	0.0003	13.1	0.0073	20.31***	11.3	0.0073	11.86***	11.4	0.0102	9.30***	5.0	0.0105	1.61**
Vision only	8.9	0.0003	8.4	0.0060		13.7	0.0079		10.5	0.0098		4.4	0.0099
Ambulatory only	23.9	0.0004	18.2	0.0084		21.4	0.0094		14.8	0.0114		27.3	0.0215
Cognitive only	28.8	0.0004	15.7	0.0079		16.2	0.0084		18.4	0.0125		30.1	0.0221
Multiple	27.5	0.0004	44.6	0.0108		37.3	0.0111		44.9	0.0160		33.2	0.0227

Note.—Weighted by gender, age, race/ethnicity, and region.

Pearson’s chi-squared (χ²) tests for differences in weighted proportions of each sample to ACS proportions.

*

p < .05;

**

p < .01;

***

p < .001.

Although sample weights are known to reduce sample selection bias (Baker et al. 2013) using observed information, these results suggest that the application of sample weights based on gender, age, race/ethnicity, and region may not sufficiently address sample selection bias by disability type. We generated new sample weights based on gender, age, race/ethnicity, and now the mutually exclusive disability types. When using these weights, proportions of each collected sample matched the ACS proportions.

Returning to the three socioeconomic outcomes, table 6 lists reweighted percentages of college education, household income, and current employment status by data source. Computed t-tests reveal that all statistically significant differences between each collected sample and ACS benchmarks remain even after reweighting to include disability type. This suggests the need to investigate more complex sample weights. There may also be unobserved characteristics that need to be addressed. A Heckman correction model could be used if panel managers were willing to share individual-level data on nonresponding panel members.

Table 6.

Reweighted Percentages of Socioeconomic Characteristics Among People With Disabilities, by Data Source

	ACS		RDD			Health Cond Panel			Gen Pop Panel			Self-Recruited Web
	%	SE	%	SE	t	%	SE	t	%	SE	t	%	SE	t
Currently employed	34.3	0.0005	38.2	0.0106	−3.08**	41.0	0.0112	−3.47**	48.9	0.0163	−6.51***	66.7	0.0227	−10.47***
Four-year college degree or higher	13.2	0.0005	29.9	0.0100	14.33***	41.3	0.0113	14.87***	25.3	0.0142	6.08***	48.7	0.0263	9.31***
Annual household income < $30k	39.4	0.0003	43.4	0.0124	2.67**	35.5	0.0110	−2.27*	44.7	0.0163	2.35***	48.3	0.0274	2.25*

Note.—Weighted by gender, age, race/ethnicity, region, and disability type.

t-Tests for differences in unweighted mean of each sample to ACS mean.

*

p < .05;

**

p < .01;

***

p < .001.

Overall, the RDD sample, which was the only collected probability-based sample, most frequently produced estimates closest to the ACS. This suggests that the expense of the RDD may have been worth it and further suggests that less expensive probability-based Internet panels could be very much worth investigating.

In contrast, the results of our self-recruited sample reflect the pitfall of convenience samples. It was most frequently the sample to produce estimates furthest from the ACS estimates. In such a convenience sample, there are many possible sources of bias (Baker et al. 2013). It is worth noting that the self-recruited sample was more concentrated in the Northeast, and respondents reported more cognitive-only disabilities. This likely reflects unevenness in the recruiting methods: the composition of the listservs and web/social media traffic that were used. These sources, in turn, likely reflect the locations and disability type-specific networks of the organizations that led the recruiting effort.

Notwithstanding the ease of recruitment, our study suggests that findings from surveys based on convenience samples should be interpreted with caution. Although the limited utility of such convenience sample surveys is well established, disability researchers resort to convenience rather than probability sampling (e.g., Vasudevan 2016; Eckstein, Sevak, and Wright 2017; Jerath et al. 2019) because identifying certain disabilities within the general population can be time- and resource-consuming. For example, surveying users of Assistive Technology (AT) in the general population requires developing and implementing a sophisticated screening protocol to identify individuals with disabilities who use specific technology to overcome their functional limitations. Pragmatically, researchers tap their professional networks, such as AT service providers, to recruit participants for such surveys (Vasudevan 2016; Jerath et al. 2019). Using river and snowball samples minimizes the need for extensive screening and verification of potential respondents.

Using prescreened panel samples may seem like a reasonable alternative and cost-effective way to identify low-incidence populations. The Internet health conditions panel did not provide significantly better estimates of people with disabilities than the general population panel. Although Qualtrics did not provide descriptive information about the frequency with which panel members experienced any of the conditions for which they were prescreened, it seems likely that most of the health conditions experienced were unrelated to our operationalization of functional disabilities. For example, conditions such as psoriasis, infertility, or sinusitis can be considered medical ailments although they have little or no impact on a person’s ability to see, walk, hear or remember things. Other vendors may prescreen panel members using different criteria to identify health conditions. As such, further research may be warranted on the use of specialized or targeted panels for disability research.

4. FURTHER DISCUSSION

Although not immediately evident in our results, there are several other issues worth discussing. First, allowing proxy response is often used to facilitate the survey participation of people with disabilities (Markesich et al. 2019). As summarized in table 1, the two Internet panels did not allow for the use of proxies. It is not clear whether persons who may need proxies would opt into an Internet access panel. If so, they may be disadvantaged in what could be a race to accept invitations. It may be necessary to oversample such individuals, and the burden would likely fall to researchers to try to disentangle whether respondents were, in fact, proxies. Proxy responders may be unlikely to self-identify if they are access panel members, as panel administrators typically take great care to verify the identity of their panelists (e.g., Qualtrics’ use of a double verification process is presented as a selling point for researchers).

Second, as summarized in table 1, incentives of unknown value were offered to both Internet panel samples and not to the RDD or self-recruited web sample. Internet panel members who are managed by companies such as Qualtrics, YouGov, and others often receive incentives to complete surveys. These may include cash payments, free downloads, and/or membership points. Although the researchers have no part in the selection or dispensation of these incentives, they could contribute to selection bias relative to researcher-recruited samples who receive either no incentive or study-specific rewards (Massey, Tourangeau, Singer, and Ye 2013; McGonagle and Freedman 2017).

Third, the context of a survey—the way a survey is titled, billed, or motivated and its primary content—may substantially impact the participants it attracts (Mercer, Lau, and Kennedy 2018). As noted earlier, stakeholders who learned of our original national employment and disability survey were eager to participate, and we created the self-recruited web sample to answer this demand. Respondents knew in advance that the survey pertained to disability and employment, a fact which likely contributes to the disproportionate percentage of participants who reported being currently employed (66.4 percent among the weighted sample compared to 34.9 percent in the ACS). Similar differences occur in national federal surveys. For example, the prevalence of disability is higher in the NHIS (Lauer and Houtenville 2018) and the BRFSS than in the ACS, CPS, and SIPP. This could be in part owing to the fact that known health-related surveys prime respondents’ attention to those aspects of their experience (Groves et al. 2006).

5. CONCLUSION

For many reasons, including ease of use, speed of data collection, and relatively low cost, research with people with disabilities using web-based opt-in panels offers an imperfect yet promising method for increasing our understanding of the characteristics and experiences of this population. Keeping in mind myriad limitations associated with nonprobability-based Internet samples (Baker et al. 2013), researchers interested in studying people with disabilities can, at minimum, use population benchmarks from the US Census to apply quotas or sample weights to the key demographic variables (Yeager et al. 2011), including disability type. Of course, sample weights can only address selection bias based on observable information. It is worth noting that the findings presented here are not generalizable to online or mixed-mode probability-based Internet panel samples. Ultimately, although this study was unable to include a probability-based Internet sample, they may represent a “sweet spot” in their ability to match population benchmarks at a reasonable price.

Acknowledgement

This study was supported by Kessler Foundation cooperative agreement number 797-1533-SEG-FY2014 and the US Department of Health and Human Services, Administration for Community Living, National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) (grant number, 90RTGE0001). The information contained here do not necessarily represent the policies of the US Department of Health and Human Services, and endorsement by the Federal Government should not be assumed (Edgar, 75.620 (b))

Footnotes

1

For example, the Urban Institute’s Health Reform Monitoring Survey draws a stratified random sample from the GfK KnowledgePanel^®, a probability-based Internet panel created from and based on a representative sample of US households recruited from a list of nearly all addresses in the United States, with laptop computers and Internet access provided for free to households as needed (Long et al. 2014).