Abstract

Although critical analysis of survey research is limited when reviewers and readers cannot view a study's questionnaire, access to novel questionnaires used in published research has not been systematically examined. The authors identified publications reporting the results of novel questionnaires in three medical journals (JAMA, The New England Journal of Medicine, and The Lancet) in January 2000–May 2003 and searched portable document format (PDF) versions of the studies for the complete questionnaire or a Uniform Resource Locator (URL) providing access to the questionnaire. When the questionnaire was not provided in the publication or a published URL, the authors requested it from the corresponding author in writing up to three times over a 6-week period. Of 93 publications with novel questionnaires, four printed the questionnaire in the article and three provided online access. Corresponding authors failed to provide questionnaires for 37 of 81 (46%) studies. Novel questionnaires used in published research are frequently not available to readers or researchers. Policies that improve access to novel questionnaires will allow better assessment of study results, reduce duplicated efforts, and improve authorship attribution for questionnaire design.

Because inaccessible questionnaires hinder the evaluation and interpretation of survey results (1–3), researchers have called for better access to novel questionnaires used in published research (4–6). Finding no previous studies of novel questionnaire availability, we systematically examined access to novel questionnaires used in research published in three prominent medical journals.

MATERIALS AND METHODS

A MEDLINE search (US National Library of Medicine, Bethesda, Maryland) was performed in May 2003 to identify published research reporting data from novel questionnaires as their primary outcome. The OVID search (Ovid Technologies, Inc., New York, New York) used the following terms: “survey” (keyword), “questionnaire” (keyword), and “questionnaires” (Medical Subject Headings (MeSH) term). Each term was searched separately and then combined with the Boolean expression “OR.” The search was limited to three journals (JAMA, The New England Journal of Medicine, and The Lancet) for the time period January 2000–May 2003.

Printed portable document format (PDF) versions of putative qualifying articles were examined by two investigators who independently extracted methods and results to determine which articles met inclusion criteria, including original research with a main outcome that reported results from a novel questionnaire. Questionnaires generally were classified into one of three categories. Many manuscripts stated that the questionnaires were developed for the purpose of the study; these were deemed novel. The second category included published, validated instruments with proper names, for example, The Edinburgh Postnatal Depression Scale and the Behavioral Risk Factor Surveillance System. The third category included questionnaires that were not clearly in either of the first two categories and required further investigation. In these instances, the questionnaires' references were reviewed to determine whether they cited a previously published questionnaire (nonnovel) or publications that supported the design of the study's questionnaire (novel). PDF versions of qualifying studies were manually searched independently by two investigators for publication of a complete questionnaire or publication of a Uniform Resource Locator (URL) providing access to the questionnaire.

During the summer of 2004, requests for questionnaires were sent to unique corresponding authors of studies with unpublished novel questionnaires. Only one questionnaire was requested from each author (when the same corresponding author represented more than one publication, the questionnaire from only one randomly selected publication meeting the inclusion criterion was requested). Up to three requests were sent at 2-week intervals via e-mail (or standard post when functional e-mail addresses were unavailable) to nonresponding authors. The corresponding author was asked to provide a copy of the questionnaire either by e-mail or standard post. If the corresponding author asked for further clarification as to why the questionnaire was being requested, the intent of the study was disclosed. Contact addresses were obtained from the published articles.

The study received Colorado Multiple Institutional Review Board approval (protocol 03-556).

RESULTS

Publication selection is outlined in figure 1. Ninety-three publications qualified as reporting novel questionnaire results; seven of the 93 (8 percent) provided the complete questionnaire (four in print, three online). The remaining publications not providing questionnaires were represented by 81 unique corresponding authors.

FIGURE 1

Publication selection process. URL, Uniform Resource Locator.

FIGURE 1

Publication selection process. URL, Uniform Resource Locator.

Requests for the unpublished novel questionnaires were sent to these 81 authors. Novel questionnaires could not be obtained from 37 of the 81 (46 percent) corresponding authors (table 1). We disclosed the intent of the study to 16 authors who asked why we were requesting the questionnaire; 10 of the 16 provided it. Overall, novel questionnaires used in 58 percent of studies (51 of 88) were available via either the publication (n = 7) or the corresponding author (n = 44). Novel questionnaires used in 42 percent of studies (37 of 88) were not available.

TABLE 1

Outcomes of requests for unpublished novel questionnaires (n = 81)

 No. 
Questionnaires obtained 44 54 
Questionnaires not obtained 37 46 
    JAMA (n = 48) 21 44 
    The New England Journal of Medicine (n = 11) 27 
    The Lancet (n = 22) 13 59 
Reasons why questionnaires were not provided*   
    Author never responded 19  
    Author corresponded, no reason given  
    Author reported no access to the questionnaire  
    Author reported that the questionnaire was given to collaborators only  
    Author was attempting to publish the questionnaire  
    Author referred to a questionnaire description in the article  
    Author offered to discuss the questionnaire  
    Author returned a questionnaire unrelated to the paper  
    Author returned a partial questionnaire  
    No correct author contact information was available  
 No. 
Questionnaires obtained 44 54 
Questionnaires not obtained 37 46 
    JAMA (n = 48) 21 44 
    The New England Journal of Medicine (n = 11) 27 
    The Lancet (n = 22) 13 59 
Reasons why questionnaires were not provided*   
    Author never responded 19  
    Author corresponded, no reason given  
    Author reported no access to the questionnaire  
    Author reported that the questionnaire was given to collaborators only  
    Author was attempting to publish the questionnaire  
    Author referred to a questionnaire description in the article  
    Author offered to discuss the questionnaire  
    Author returned a questionnaire unrelated to the paper  
    Author returned a partial questionnaire  
    No correct author contact information was available  
*

Refer to the supplemental table for more details. (This table is posted on the Journal's website (http://aje.oupjournals.org).)

For the 37 corresponding authors not providing questionnaires, 10 studies acknowledged funding from a US federal agency (e.g., National Science Foundation, Centers for Disease Control and Prevention, Agency for Healthcare Quality and Research, and the Office of US Foreign Disaster Assistance), including five studies acknowledging National Institutes of Health funding.

DISCUSSION

Researchers, publishers, and readers may have multiple conflicting concerns regarding access to novel questionnaires. Researchers may be reluctant to share a questionnaire because of fear that their work will be duplicated without credit. Busy researchers, with competing demands, may put off requests for questionnaires or may be unable to readily access their own work. Readers and reviewers without access to questionnaire layout and the wording of questions may lack confidence in the author's interpretation of the results.

Because the scientific process includes the replication of findings, lack of access to questionnaires limits both the critical analysis and replication of the methods. Until the era of digital publishing, the value of print space hindered the ability of journals to publish complete questionnaires. Electronic publication and archival methods have now eliminated the space limitations of the printed page.

Recently, the National Institutes of Health implemented and funded PubMed Central, a digital repository of published research. Submission by investigators and journals is voluntary but highly encouraged, with the goals of improving public access to research findings and creating a stable electronic archive (7). The National Institutes of Health hopes to use this database “to manage its research investment, monitor scientific productivity, and help set research priorities” (7, p. 1740).

Creation of a questionnaire database may serve analogous goals and address stakeholder concerns. First, a questionnaire database enables all questionnaires, not just novel ones, to be linked to studies that validated or used the questionnaire. Likewise, a history of revisions and modifications could be linked, allowing researchers easy access to important information such as what version was validated and whether recent modifications need revalidation. Most importantly, a questionnaire repository would guarantee full access to research methods independent of author correspondence.

The digital information revolution transforming academic publishing and archival methods makes inaccessible data measurement instruments unacceptable. Our study demonstrates that a reasonable attempt by a reader to obtain an unpublished questionnaire used in research published in prestigious medical journals (three written requests to the corresponding author over a 6-week period) frequently fails.

To improve research transparency and to avoid duplicated questionnaire development efforts, we call on scholarly journals to require that novel questionnaires be submitted with manuscripts for prepublication peer review and to mandate questionnaire publication in one of the following: 1) the print journal, 2) a journal online supplemental repository, or 3) other digital information repositories, such as Google Base (8) or a questionnaire database administered by a national library. Guaranteeing access to research questionnaires will foster improved pre- and postpublication peer review and promote optimal utilization of these data measurement instruments for future research.

Support for this study was provided in part by National Institutes of Health grant K-07 CA92550 (Dr. Dellavalle) and by grant 5 D14HP00153, a Faculty Development in Primary Care Health Services Research Award (Dr. Schilling).

The authors thank Dr. Kristie McNealy and Lauren Heilig for their assistance with this study.

This work was presented orally at The Fifth International Congress on Peer Review and Biomedical Publication on September 15, 2005, in Chicago, Illinois.

Conflict of interest: none declared.

References

1.
Hunt
SD
Sparkman
RD
Wilcox
JB
The pretest in survey research. Issues and preliminary findings
J Market Res
 , 
1982
, vol. 
19
 (pg. 
269
-
73
)
2.
Martinson
BC
Anderson
MS
De Vries
R
Scientists behaving badly
Nature
 , 
2005
, vol. 
435
 (pg. 
737
-
8
)
3.
Boynton
PM
Greenhalgh
T
Hands-on guide to questionnaire research. Selecting, designing, and developing your questionnaire
BMJ
 , 
2004
, vol. 
328
 (pg. 
1312
-
15
)
4.
Olsen
J
Epidemiology deserves better questionnaires. IEA European Questionnaire Group International Epidemiological Association
Int J Epidemiol
 , 
1998
, vol. 
27
 pg. 
935
 
5.
Wilcox
AJ
The quest for better questionnaires
Am J Epidemiol
 , 
1999
, vol. 
150
 (pg. 
1261
-
2
)
6.
Gallacher
J
Epidemiology deserves better questionnaires. (Letter)
Int J Epidemiol
 , 
1999
, vol. 
28
 pg. 
804
 
7.
Steinbrook
R
Public access to NIH-funded research
N Engl J Med
 , 
2005
, vol. 
352
 (pg. 
1739
-
41
)
8.
Butler
D
Google makes data free for all
Nature
 , 
2005
, vol. 
438
 (pg. 
400
-
1
)

Author notes

Editor's note:An invited commentary on this article is published on page 1145.