Reporting Participation in Epidemiologic Studies: A Survey of Practice

Abstract

Self-selection bias may threaten the internal validity of epidemiologic studies. Studies with a low level of participation are particularly vulnerable to this bias, and commentators note apparent declines in participation in recent years. The authors therefore conducted a retrospective review to survey the practice of reporting participation in epidemiologic studies, to assess changes in participation over time, and to evaluate the impact of increased biologic specimen collection on participation. The authors abstracted selected study characteristics from 355 peer-reviewed, original, analytic-epidemiology research articles published from January 1 to April 30, 2003, in 10 high-impact general epidemiology, public health, and medical journals. At least some information regarding participation was provided in 59% of cross-sectional studies, 44% of case-control studies, and 32% of cohort studies. Participation appears to have declined during 1970–2003 for all study designs. Participation declined most steeply for controls in population-based, case-control studies (−1.86% per year, 95% confidence interval: −3.03, −0.69), with steeper declines after 1990. Proportionately more studies collected biologic specimens over time, particularly for cohort and case-control study designs (p_trend = 0.06 and 0.03, respectively), yet participation was reported separately for the biologic specimen study component in only 27% of studies. The authors conclude that epidemiologists need to address declining participation and to report participation consistently, including for biologic specimen collection.

blood specimen collection, case-control studies, cohort studies, cross-sectional studies, epidemiologic methods, patient participation

CI, confidence interval

The internal validity of epidemiologic studies can be threatened by self-selection bias resulting from differences between those who participate in a study and those who do not (1). Epidemiologic studies with low levels of participation may be more vulnerable to self-selection bias than those with high participation (2). Although it is widely perceived that participation has declined in recent years (3), assessments of the time trends for participation have been hampered by a lack of detailed, consistent reporting of participation in the literature (4, 5). Further, epidemiologic studies increasingly include biologic specimens to measure exposure, disease, or surrogates, with unknown impact on participation.

We therefore surveyed the practice of reporting participation in original, analytic-epidemiology research articles to evaluate changes in participation and in the collection of biospecimens over time and to assess the proportion of studies reporting participation separately for biologic specimen collection.

MATERIALS AND METHODS

Data sources and collection

We conducted a retrospective review of original, analytic-epidemiology research articles published in major epidemiology, public health, and general medical journals. Journals that published peer-reviewed, original research articles in English and were listed in the 2003 Science Edition of Journal Citation Reports (The Thompson Corporation, Philadelphia, Pennsylvania) were eligible for inclusion. We selected 10 journals for review: seven journals containing the words “epidemiology” or “public health” in the title with a 2003 impact factor of at least two, and three general medical journals with the highest 2003 impact factor (table 1). We excluded review journals (e.g., Annual Review of Public Health) and specialty journals (e.g., Infection Control and Hospital Epidemiology, Public Health Nutrition).

TABLE 1.

Journals selected for inclusion in the review of 355 original, analytic-epidemiology research articles published from January 1 to April 30, 2003

Journal title	2003 impact factor	Selection criteria					Published issues (no.)	Analytic- epidemiology articles
				Journal title contains						General medical journal
				“Epidemiology”	“Public health”				No.		%
American Journal of Epidemiology	4.49	x			8	57	16.1
American Journal of Public Health	3.36		x		4	36	10.1
Annals of Epidemiology	2.35	x			4	29	8.2
Cancer Epidemiology, Biomarkers & Prevention	4.72	x			4	41	11.5
Epidemiology	4.22	x			2	20	5.6
International Journal of Epidemiology	3.29	x			2	27	7.6
Journal of Clinical Epidemiology	2.23	x			4	12	3.4
Journal of the American Medical Association	21.46			x	16	36	10.1
Lancet	18.32			x	17	54	15.2
New England Journal of Medicine	34.83			x	17	43	12.1

We abstracted selected study characteristics from original, analytic-epidemiology research articles published in the 10 journals from January 1 to April 30, 2003, representing one third of the issues published in each journal during that year. A total of 355 original, analytic-epidemiology research articles were identified and included in this review. References for the abstracted articles used as data sources are available in the Appendix posted on the Journal's website (http://aje.oxfordjournals.org/). For all studies, we recorded the study design (1); years of data collection; location (US, other); any details on participation; and whether the study was multicenter, relied exclusively on records for either the main exposure or outcome, referenced another report for methodology, or collected a biologic specimen (e.g., blood, urine, stool, cervical swab). Case-control study designs were determined by the method used to recruit controls (population based; hospital based; family, friend, or neighborhood based; nested within a cohort; or unknown). For prospective cohort studies, we also recorded information on loss to follow-up. If a study reported the collection of a biologic specimen, we recorded any information on participation for the biologic specimen component.

Various reporting practices and incomplete information complicate the assessment of participation rates from published reports (4, 6). For this retrospective review, we used standard definitions from the American Association for Public Opinion Research (7). A “response rate” (more properly termed “response proportion”) was defined as the number of participants divided by the sum of the numbers of participants, nonparticipants (including refusals and noncontacts), and persons of presumed but unconfirmed eligibility (7). A “cooperation rate” (again, more properly termed “cooperation proportion”) was defined as the number of participants divided by the number eligible that were ever contacted (7). We use the term “participation rate” broadly, that is, to refer to either cooperation rates or response rates, because many authors fail to specify the definition of the participation rate they report. If an article reported both a response rate and a cooperation rate, we used the cooperation rate in our analyses of “participation rates.”

For case-control studies, we judged the reporting of participation to fall into one of three categories: 1) “no information” (no information on participation was reported), 2) “some information” (a measure of participation was reported but not clearly defined), or 3) “adequate information” (both the response and cooperation rates (or sufficient information to estimate these measures) were reported). For cohort, cross-sectional, and ecologic studies, we judged the reporting of participation to fall into one of two categories because of a general lack of information regarding the sampling frame: 1) “not reported” (no information on participation was reported) or 2) “reported” (at least some information on participation was reported). When judging the reporting of participation, we did not require specific language, thus allowing for flexibility in the manner of reporting. We ascertained participation information only from the surveyed reports; we did not consider participation information from additional reports referenced for methodology.

Statistical analyses

We computed simple descriptive statistics to measure the proportions of studies reporting participation according to study design and tested differences in the quality and proportion of reporting by Fisher's exact test. To evaluate potential changes in participation rates over time, we constructed linear regression models, with the predictor variable defined as the first year of data collection (baseline) for cohort studies or as the median year of data collection for case-control and cross-sectional studies (1970–2003). Because previous research has suggested that participation rates may have declined significantly only recently (8), we conducted an additional analysis restricted to those studies with the first year of data collection (cohort studies) or the median year of data collection (case-control and cross sectional studies) after 1990. We considered the journal type (general medical vs. epidemiology or public health), location (US, international, or US and international), and multicenter design (yes, no) as potential confounding factors in the analyses. Inclusion of location or multicenter design in the linear regression models did not change the parameter estimates by 10 percent or greater; therefore, we present results from analyses adjusted by journal type.

We compared the proportions of studies that collected biologic specimens over time based on the first year of data collection (before 1990, 1990–1994, 1995 or later) using the Cochran-Armitage test for trend. We considered the proportion of studies reporting participation separately for the biologic specimen study component and computed the median and range for the reported biologic specimen participation rates according to study design. Tests of statistical significance were two sided, with an α level of 0.05, and are reported with one significant digit. Statistical analyses were conducted by use of SAS, version 9.1, software (SAS Institute, Inc., Cary, North Carolina).

RESULTS

We abstracted information from 355 original, analytic-epidemiology research articles published from January 1 to April 30, 2003, in seven general epidemiology and public health journals (63 percent of the articles) and three general medical journals (37 percent of the articles) (table 1). The articles reported results from 154 cohort, 107 case-control, 86 cross-sectional, and six ecologic study designs, as well as from two case series.

Among the reports from case-control studies, the authors reported at least some participation data in 47 of 107 (44 percent) articles, giving a median participation rate of 84 percent for cases and 74 percent for controls (table 2). Few case-control study articles (16 percent) reported adequate information to compute both the response and cooperation rates. The quality of reporting varied significantly by the specific design of the case-control study (p = 0.002), with the most reporting of participation information in population-based and family/friend/neighborhood-based case-control studies (74 percent and 57 percent of articles, respectively). For more than 70 percent of hospital-based, nested, and unknown-design case-control studies, the authors reported no participation information. Generally, the reported participation rates were slightly higher in hospital-based and nested case-control studies than in population-based, case-control studies.

TABLE 2.

Reporting of participation in analytic-epidemiology case-control studies, based on the review of original, analytic-epidemiology research articles published from January 1 to April 30, 2003, in 10 high-impact general epidemiology, public health, and medical journals

	Adequate information			Some information			No information			Median participation*
													Cases			Controls
	No.	%	No.	%	No.	%	%	Range	%	Range
Total case-control studies (n = 107)	17	16	30	28	60	56	84	24–100	74	15–99
By study design
Population based (n = 34)	11	32	14	41	9	26	84	44–99	74	41–88
Hospital based (n = 33)	2	6	7	21	24	73	92	74–99	86	60–99
Family/friend/neighborhood (n = 7)	2	29	2	29	3	43	73	66–87	63	44–90
Nested (n = 24)	1	4	6	25	17	71	88	60–100	—†
Unknown (n = 9)	1	11	1	11	7	78	52	24–80	—†

*

Participation rates as reported in articles that provided at least some information; if both the response rates (participants/total eligible + presumed eligible) and the cooperation rates (participants/contacted) were reported, the cooperation rate was used in computing the median participation.

†

—, only one study provided information on participation for controls.

The reporting of participation varied significantly among cohort, nested case-control, and cross-sectional studies (p < 0.0001) (table 3), with participation provided most often in cross-sectional studies and least often in nested case-control studies (59 percent and 21 percent of articles, respectively). Participation information was not provided in any of the retrospective cohort studies, probably because 21 of the 23 (91 percent) studies were record based (data not shown). Information on loss to follow-up was provided in 53 of 120 (44 percent) prospective cohort studies, with a median loss to follow-up of 8.5 percent (data not shown). A total of 140 of 355 (39 percent) articles referenced another report for methodology, but there were no significant differences in the proportion of studies reporting participation information among those articles that did or did not reference another report for methodology (data not shown).

TABLE 3.

Reporting of participation in analytic-epidemiology cohort, nested case-control, and cross-sectional studies, based on the review of original, analytic-epidemiology research articles published from January 1 to April 30, 2003, in 10 high-impact general epidemiology, public health, and medical journals

	Reported			Not reported			Median participation*
	No.	%	No.	%	%	Range
Total cohort studies (n = 154)	49	32	105	68	80	20–100
By study design
Prospective (n = 120)	45	38	75	63	81	20–100
Ambispective (n = 7)	3	43	4	57	75	50–80
Retrospective (n = 23)†	0	0	23	100
Case-cohort (n = 4)	1	25	3	75	—‡
Nested case-control studies (n = 24)	5	21	19	79	87	37–99
Cross-sectional studies (n = 86)	51	59	35	41	74	28–100

*

For cohort and nested case-control studies, we recorded information on the reporting of baseline participation rates in the cohort.

†

No retrospective studies provided information on participation.

‡

—, only one study provided information on participation.

Participation rates in the abstracted reports declined gradually over time (table 4). In case-control studies, participation rates during 1970–2003 changed –1.18 percent (95 percent confidence interval (CI): −2.33, −0.02) and −1.49 percent (95 percent CI: −2.94, −0.05) per year for cases and controls, respectively. Population-based, case-control studies suffered worse declines than did hospital-based, case-control studies for both cases (−1.32 percent vs. −0.76 percent per year) and controls (−1.86 percent vs. −0.09 percent per year). Limiting our analysis to those population-based, case-control studies with a median year of data collection after 1990 revealed that participation rates declined more sharply during 1991–2003 than during 1970–1990 for both cases (−3.33 percent (95 percent CI: −9.22, 2.56) vs. −0.24 percent (95 percent CI: −2.59, 2.12)) and controls (−5.15 percent (95 percent CI: −9.23, −1.10) vs. −2.70 percent (95 percent CI: −5.50, 0.09)) (data not shown). In cohort and cross-sectional studies, participation rates during 1970–2003 changed −0.54 percent (95 percent CI: −1.33, 0.24) and −0.67 percent (95 percent CI: −1.91, 0.56) per year, respectively. Adjustment of the linear regression models for location (US, international, or US and international) or multicenter design (yes, no) did not materially alter the parameter estimates (data not shown).

TABLE 4.

Changes in participation during 1970–2003* by study design, based on the review of original, analytic-epidemiology research articles published from January 1 to April 30, 2003, in 10 high-impact general epidemiology, public health, and medical journals

	No.	β1†	95% confidence interval	p value
Cohort studies	47	−0.87	−2.01, 0.27	0.1
Case-control studies‡
Cases	37	−1.18	−2.33, −0.02	0.05
Population based	24	−1.32	−2.54, −0.10	0.04
Hospital/clinic based	8	−0.76	−5.60, 4.07	0.7
Family/friend/neighborhood§	3
Unknown§	2
Controls	33	−1.49	−2.94, −0.05	0.04
Population based	22	−1.86	−3.03, −0.69	0.003
Hospital/clinic based	7	−0.09	−8.02, 7.83	0.9
Family/friend/neighborhood§	3
Unknown§	1
Cross-sectional studies	50	−1.05	−2.39, 0.30	0.1

*

In cohort studies, baseline participation: time = first year of data collection; in case-control and cross-sectional studies, participation: time = median year of data collection. Excludes studies that did not report the year the study was conducted (two prospective cohort studies, three case-control studies, and one cross-sectional study).

†

Model: participation (%) = β₀ + β₁ × year + β₂ × journal type; adjustment for location (US, international, US and international) or multicenter design (yes, no) did not materially alter the parameter estimates.

‡

Excludes all nested case-control studies.

§

Insufficient data to assess changes in participation over time.

A total of 134 of 355 (38 percent) articles reported the collection of biologic specimens to measure exposure or disease. The proportion of cohort and case-control studies that collected biologic specimens increased over time (p_trend = 0.06 and 0.03, respectively) (table 5). Approximately one third of the cohort, nested case-control, and cross-sectional studies reported participation separately for the biologic specimen component of the study (table 6). In contrast, only 22 percent of the population-based, case-control studies and 13 percent of the hospital-based, case-control studies reported participation separately for the biologic specimen component of the study. The overall median participation rate among the studies that reported information on participation was 87.5 percent. Participation in the biologic specimen component tended to be highest in cohort studies (median, 99.5 percent).

TABLE 5.

Changes over time* in the proportion of analytic-epidemiology studies that collected biologic specimens, based on the review of original, analytic-epidemiology research articles published from January 1 to April 30, 2003, in 10 high-impact general epidemiology, public health, and medical journals

	Before 1990			1990–1994			1995 and later			ptrend
	No.	%	No.	%	No.	%
Cohort studies (n = 145)	18/65	28	11/31	35	22/49	45	0.06
Case-control studies (n = 90)	11/31	35	14/27	52	12/20	63	0.03
Cross-sectional studies (n = 84)	5/11	45	1/20	5	11/53	21	0.3

*

Time = first year of data collection; date was missing for nine cohort studies, 17 case-control studies, and two cross-sectional studies.

TABLE 6.

Reporting of participation for the biologic specimen component within 134 analytic-epidemiology studies that collected biologic specimens, based on the review of original, analytic-epidemiology research articles published from January 1 to April 30, 2003, in 10 high-impact general epidemiology, public health, and medical journals

	Reported			Not reported			Median participation proportion
	No.	%	No.	%	%	Range
All studies (n = 134)	36	27	98	73	87.5	36–100
Total cohort studies (n = 56)	20	36	36	64	99.5	70–100
Nested case-control studies (n = 12)	4	33	8	67	58*	36–83*
					70†	47–83†
Other case-control studies (n = 47)	6	13	41	87	86*	82–100*
					83†	76–98†
By study design
Population based (n = 9)	2	22	7	78
Hospital based (n = 24)	3	13	21	88
Family/friend/neighborhood (n = 7)	1	14	6	86
Unknown (n = 7)	0	0	7	100
Cross-sectional studies (n = 18)	6	33	12	67	90.5	42–99
Case series (n = 1)	0	0	1	100

*

Median participation proportion among cases.

†

Median participation proportion among controls.

DISCUSSION

Our retrospective review demonstrates that a surprising number of case-control (56 percent), cohort (68 percent), and cross-sectional (41 percent) studies recently published in the seven highest-impact general epidemiology and public health journals, and that the three highest-impact general medical journals failed to report any information regarding participation rates. Data from those studies that did report participation rates demonstrated a decline in participation over time. The sharpest declines occurred in population-based, case-control studies, particularly for controls. The proportion of epidemiologic studies collecting biologic specimens has increased significantly, but it is difficult to discern the impact of biologic specimen collection on participation rates because authors seldom report participation rates separately for the biologic specimen component of the study.

Although some differences by study design in the level of participation are to be expected, such as higher participation among hospital than population controls, the reasons for the differences in the frequency of reporting participation are unclear. Some variation in the reporting may be accounted for by differences by study design in the risk of bias resulting from nonparticipation. For example, case-control and cross-sectional studies are more subject than are prospective cohort studies to self-selection bias that is related to both exposure and disease and thus threatens the internal validity of study results. Nevertheless, the surprisingly low level of reporting participation in all studies regardless of design is of concern.

The surprisingly low level of reporting participation that we observed in analytic-epidemiology studies generally agrees with a previous review of cohort and case-control studies, which found that 34 of 73 (47 percent) studies reported no information on participation (9), and with a previous review of observational longitudinal studies, which demonstrated that nine of 49 (18 percent) reviewed studies reported the number of individuals who consented or refused at the beginning of the study and that 25 of 49 (51 percent) studies reported the numbers at each stage (i.e., loss to follow-up) (5). The lack of consistent, detailed reporting of participation in epidemiologic studies has been noted previously (4), yet standardized reporting guidelines for observational epidemiologic studies have not been adopted. In contrast, standardized reporting guidelines for clinical trials are extensive (10, 11), and implementation of these guidelines has improved the reporting of essential methodological elements (12). Guidelines for the reporting of nonrandomized interventions (13) and meta-analyses (14, 15) also have been published and adopted. In 2002, Olson et al. (16) proposed a rigorous set of standards for the reporting of participation in case-control studies, including complete enumeration for each participant recruitment method of sampling units that were ineligible or for whom eligibility could not be determined and enumeration of respondents who were eligible and refused or eligible and participated. Detailed reporting of subject recruitment methods and the level of participation at each step as proposed by Olson et al. (16) may be particularly important if there are differences between refusals and noncontacts (8, 17, 18). On the other hand, the current data suggest that even a modest increase in reporting basic participation information would constitute a major improvement.

This retrospective review of 355 articles from 10 journals covering a wide range of disciplines demonstrates that there has been a decline in participation rates over time, particularly for controls in population-based, case-control studies. Our analysis also suggests that the decline in participation rates may have accelerated in recent years. The reasons for the decline are not well understood, although changes over time in study design, methods for recruitment, and societal factors that may influence participation are likely explanations.

The decline in participation that we observed in epidemiologic research is consistent with that observed in previous survey research. In the University of Michigan's Survey of Consumer Attitudes, a monthly telephone survey conducted for more than 30 years, participation declined very slightly during 1979–1991 but much more sharply during 1991–2002 (−0.2 percent vs. −1 percent per year), and increased efforts to make contact by telephone have been required in recent years (8). A previous analysis of participation in 82 case-control studies in the United States and Canada suggested a nonsignificant change in participation for controls of −0.44 percent per year based on a median year of data collection during 1972–1996 (6), which is consistent with our results.

Declining participation and the vulnerability of studies with low participation to self-selection bias increase the importance of understanding the determinants of response in various study designs. Research into this topic conducted over many years suggests that there are many factors that affect participation, including the method of recruitment (4, 19); various family and medical history or exposure factors, especially those involved with the topic under study (18, 19) and including disease status (6, 17); questionnaire structure (18, 20, 21); and method and number of contacts (8, 18, 21–24). Conflicting results of the effect of other factors on participation have been reported, including age (4, 6, 17–19, 25), sex (4, 6, 18, 24, 25), race (17, 19, 25), education (19), and the use of incentives (8, 21, 23–25). It is likely that some differences in the effects of various factors on participation can be accounted for by differences in study design or the risk factors and disease under study. Nevertheless, clarification of the determinants of response, particularly in the current environment of increased number of telephone lines per household, cellular telephones, and the use of mechanisms (e.g., caller identification) to screen telephone calls, is needed.

The increased collection of biospecimens over time likely reflects recent advances in molecular techniques, primarily the use of biomarkers, which have necessitated changes in the methods typically used in epidemiologic studies for the assessment of both exposure and disease. Based on the small proportion of studies we reviewed that did report participation separately for the biologic specimen collection study component, participation appears to be relatively high. A more detailed understanding of the impact of biologic specimen collections on study participation will require more information in the published literature. For studies in which the biologic specimen collection is optional, reporting of participation separately for the biologic specimen component should require very few brief sentences (26). It may be more complicated, however, to understand the impact of biologic specimen collection for studies in which collection is a mandatory component of the study. The exceptionally high median participation rate of 99.5 percent reported for the biologic specimen component within cohort studies suggests that provision of a biospecimen was a criterion for inclusion in many follow-up studies.

Our survey concentrated on participation in mainstream epidemiologic research, as published in high-impact general epidemiology, public health, and major medical journals. These journals are likely to represent the highest quality of epidemiology research in the literature; a review of articles in other journals could generate different results. Our large sample size enabled us to provide a detailed assessment of the current practice of reporting participation by study design and time period. For the less-common study designs, including family/friend/neighborhood-based case-control studies, ecologic studies, and case series, additional data would be needed to quantify changes in the participation in these studies.

Our research and previous literature on this topic demonstrate that epidemiologists ought to be concerned about the lack of consistent reporting of participation. Readers, reviewers, editors, and authors all need to insist that at least a minimum amount of information regarding participation be reported in all published epidemiologic research. Reporting participation has become increasingly necessary in the current environment of declining participation rates and increasing use of biologic specimens. Although low participation itself does not necessarily compromise the internal validity of a study, information regarding participation is essential for readers to understand and to assess study results (27). The adoption of standardized reporting guidelines for observational epidemiologic studies could improve the current practice of reporting epidemiologic research and could ultimately stimulate improvements in the methods of recruiting study participants and the research itself.

Conflict of interest: none declared.