Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography

Abstract

Background Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies.

Methods Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google®. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies.

Results A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent.

Conclusion A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.

Observational studies, epidemiological studies, quality, bias, checklist, scales

Introduction

Systematic reviews identify, appraise and synthesize evidence from multiple studies of the same research question, and can be applied to diverse topics in medical research, including the effects of health-care interventions, the accuracy of diagnostic tests and the relationship between risk factors and disease. Meta-analyses, often contained within systematic reviews, offer a means of quantitatively summarizing the body of evidence identified. The strengths and limitations of systematic reviews and meta-analyses have been well established for randomized clinical trials, largely through the efforts of The Cochrane Collaboration. Although they have been used in parallel for observational epidemiological studies, such as cohort, case-control and cross-sectional studies, considerably less attention has been paid to their methodology in this area of application.

A systematic review should follow a protocol in order to minimize bias and ensure that the findings are reproducible. A key source of potential bias in a meta-analysis is bias due to limitations in the original studies contained within it. For example, a review of case-control studies of oral contraceptives and risk of rheumatoid arthritis found exaggerated effects in hospital-based control groups compared with population-based control groups¹ whilst a review of case-control studies investigating the impact of sunlight exposure on skin cancer identified an important difference between study results when subjects or interviewers were blinded (or not) to skin cancer status.² A large prospective study of the association between C-reactive protein and coronary heart disease obtained odds ratios varying from 2.13 to 3.46 with different degrees of adjustment for confounding variables.³

An important component of a thorough systematic review is therefore an evaluation of the methodological quality of the primary research. Numerous tools have been proposed for evaluation of methodological quality of observational epidemiological studies. A comprehensive study of tools for assessing non-randomized intervention studies in health care (excluding case-control studies) identified 193 tools, including several that could also be used for assessing non-intervention studies.⁴ A large-scale review of tools for grading the quality of research articles and rating the strength of bodies of evidence identified 17 tools for grading evidence from observational study designs,⁵ although it did not include some of the key tools identified in previous reviews. More recently, Katrak and colleagues⁶ reviewed 121 critical appraisal tools for allied health research, including physiotherapy, occupational and speech therapy and found a number of problems. All of these reviews have generally concluded that there is currently no agreed ‘gold standard’ appraisal tool; that the majority of tools did not undergo a rigorous development process; and that there are many tools from which to choose. Consequently, to our knowledge, no tool has been adopted for widespread use within systematic reviews. In addition, none of these reviews sought to identify all tools for assessing observational epidemiological studies.

‘Quality’ is an amorphous concept. A convenient interpretation is ‘susceptibility to bias’, although it is not uncommon for aspects of study conduct that are not directly associated with bias to be included in a quality assessment. For example, study size, whether or not a power calculation was performed, and ethical approval might be considered aspects of quality, but are, in their own right, not potential causes of bias. Our main objective was to seek tools to assess susceptibility to bias, but we do not draw a clear distinction between quality in bias, reflecting the lack of a distinction in much of the published literature.

It is important, however, to distinguish between quality of reporting and quality of what was actually done in the design, conduct and analysis of a study. A high-quality report ensures that all relevant information about a study is available to the reader, but does not necessarily reflect a low susceptibility to bias.¹ Factors such as the peer-review process, editorial policy or journal space restrictions may preclude detailed reporting and so make it difficult to assess inherent biases. A number of consensus statements have encouraged higher quality of reporting, including recommendations for reporting systematic reviews (QUOROM),⁷ randomized trials (CONSORT),⁸ studies of diagnostic tests (STARD),⁹ meta-analyses of observational studies (MOOSE)¹⁰ and observational epidemiological studies (STROBE).^11,,12 These are aimed at authors of reports, not at those seeking to assess the validity of what they read.

This study provides an annotated bibliography of tools specifically designed to assess quality or susceptibility to bias in observational epidemiological studies, obtained from a comprehensive search of the published literature and of the Internet. It follows the approach of a previous review of tools to assess quality of randomized controlled trials,¹³ and attempts to identify whether there is an existing tool that could be recommended for widespread use.

Methods

Inclusion criteria

To be included in the review, a tool was defined as any structured instrument aimed at aiding the user to assess quality or susceptibility to bias in observational epidemiological studies (cohort, case-control and cross-sectional studies). Tools were placed in one of the following three categories defined below: scales, simple checklists or checklists with a summary judgement. Scales result in a summary numerical score, typically derived as a sum of scores for several items. Checklists consisted of only a list of items, whilst checklists with a summary judgement were checklists that also resulted in an overall qualitative assessment about the study's quality, such as ‘high’, ‘medium’ or ‘low’. These tools may have been developed for use in critical appraisal or in systematic reviews, and may have been developed for general use or use in a specific context. Articles that provided general narrative guidance only or were without an explicit scale or checklist were excluded.

Search methods

Three electronic databases (MEDLINE, EMBASE and Dissertation Abstracts up to March 2005) were searched using full text and MeSH terms to identify articles discussing observational epidemiological study designs, including ‘cohort studies’, ‘case-control studies’, ‘cross-sectional studies’ and ‘follow-up studies’. Where possible, all terms were included as full text, with truncation used where possible to capture variation in the terminology. The search was not limited to the English language, nor restricted by any other means.

In order to capture tools posted on Internet websites, we conducted an Internet search using the Google® search engine¹⁴ during March 2005. Searches were conducted using several combinations of the following search terms: ‘tool’, ‘scale’, ‘checklist’, ‘validity’, ‘quality’, ‘critical appraisal’, ‘bias’ and ‘confounding’. The first 300 links identified by each separate search were investigated. Reference lists of published articles were examined to identify additional sources not identified in the database searches.

Inclusion criteria

Articles or websites were included if they described a tool suitable for assessing quality of observational epidemiological studies. Abstracts were scrutinized for suitability before obtaining the full text of all relevant articles. Where more than one tool was published within the same article or website (for example, independent tools for assessing cohort and case-control study designs published within the same article or website), these were included as separate quality assessment tools. Published reports were used in preference to web sites for tools reported in both formats. Care was taken not to include the same tool twice.

Data extraction

A data extraction form was developed and piloted and included information about the type of study addressed by the tool, number of items, scoring system, description of the development process, whether the tool was developed for generic use in systematic reviews, single use in a specific systematic review or for critical appraisal, and whether the tool was proposed for future use. Data extraction was performed by two authors (SS and IT) with differences of opinion resolved by discussion or by the third author (JH). Items in tools were classified into domains that covered key potential sources of bias. The selection was strongly influenced by the ‘STrengthening the Reporting of OBservational studies in Epidemiology’ (STROBE) guidelines for reporting observational epidemiological studies. These guidelines for reporting case-control, cohort and cross-sectional studies were developed by an international collaboration of epidemiologists, statisticians and journal editors. Although not a tool for assessing the quality of primary studies, they provide a useful indication of the essential information needed to appraise the conduct of such studies. Table 1 shows how the domains and criteria were used to evaluate tool content.

Table 1

Domains and criteria for evaluating each tool's content

Domain	Tool item must address
Methods for selecting study participants	Appropriate source population (cases, controls and cohorts) and inclusion or exclusion criteria
Methods for measuring exposure and outcome variables	Appropriate measurement methods for both exposure(s) and/or outcome(s)
Design-specific sources of bias (excluding confounding)	Appropriate methods outlined to deal with any design-specific issues such as recall bias, interviewer bias, biased loss to follow or blinding
Methods to control confounding	Appropriate design and/or analytical methods
Statistical methods (excluding control of confounding)	Appropriate use of statistics for primary analysis of effect
Conflict of interest	Declarations of conflict of interest or identification of funding sources

Wherever possible, we have attempted to demonstrate weighting within checklists and scales by including the total number of items for a checklist and the number of these items allocated to a particular quality domain. For scales, we have included the total maximum raw score for each scale and the possible total score by domain (although most scales do not address all of the domains in Table 1). A few of the tools use extremely complicated assessment and scoring systems, and for these we have reported the total raw score and the maximum item score by domain.

Results

A total of 86 tools were included in the review, 62 identified from the electronic database search (72%) and a further 24 from the Internet search (28%). An overall summary of the main tool characteristics is presented in Tables 2–4 and more detailed information in Tables 5–7.

Table 2

Summary results comparing identified tools by type

Tool characteristics	Simple checklists(n = 41)	Simple checklists with additional judgement (n = 12)	Scales (n = 33)	Total (n = 86)
Source
Electronic database	21 (51%)a	9 (75%)	32 (97%)	62 (72%)
Internet	20 (49%)	3 (25%)	1 (3%)	24 (28%)
	100%	100%	100%	100%
Tool purpose
Single use in a specific context	3 (7%)	4 (33%)	22 (67%)	29 (34%)
Generic tool for systematic reviews	8 (20%)	3 (25%)	2 (6%)	13 (15%)
Critical appraisal tool	22 (54%)	4 (33%)	5 (15%)	31 (36%)
Ambiguous (unable to allocate above categories)	8 (20%)	1 (8%)	4 (12%)	13 (15%)
	41 (100%)	12 (100%)	33 (100%)	86 (100%)
Development
Development described	21 (51%)	7 (58%)	18 (55%)	46 (53%)
Future use
Proposed for future use	38 (93%)	8 (67%)	14 (42%)	60 (70%)

^aPercentages subject to rounding error.

Table 3

Summary results comparing identified tools by content

	Simple checklists (n = 41)	Simple checklists with additional judgement (n = 12)	Scales (n = 33)	Total (n = 86)
Tool content
Number of items
• Range	3–36	4–32	4–35
• Mean	13.4	15.2	12.6
Maximum raw score range (scales only)	NA	NA	4–72
Appropriate methods for selecting study participants % (range)	39; 95%a (1–10)	11; 92% (1–6)	29; 88% (1–26.4)	79 (92%)
Appropriate methods for measuring exposure and outcome variables % (range)	36; 88% (1–10)	12; 100% (1–8)	26; 79% (1–22)	74 (86%)
Appropriate design-specific sources of bias (excluding confounding) n; % (range)	36; 88% (1–6)	11; 92% (1–10)	27; 82% (1–8)	74 (86%)
Appropriate methods to control confounding n; % (range)	34; 83% (1–5)	12; 100% (1–3)	21; 64% (1–12)	67 (78%)
Appropriate statistical methods (primary analysis of effect but excluding confounding) n; % (range)	34; 83% (1–8)	8; 67% (1–3)	24; 73% (1–20)	66 (78%)
Conflict of interest n; % (range)	1; 2% (1)	1; 8% (1)	1; 3% (1)	3 (4%)

Note: For checklists, the range represents items; for scales, it represents available raw scores.

^aPercentages subject to rounding error.

Table 4

Distribution of tools by epidemiological study design addressed

Case-control	Cohort	Cross-sectional	Simple checklists n (%)	Simple checklists with a judgement n (%)	Scales n (%)	Total n (%)
Y	N	N	9 (22)	2 (17)	5 (15)	16 (19)
Y	Y	N	15 (36)	6 (50)	7 (21)	28 (32)
Y	Y	Y	4 (10)	1 (8)	8 (24)	13 (15)
N	Y	N	11 (27)	2 (17)	10 (30)	23 (27)
N	N	Y	2 (5)	1 (8)	3 (9)	6 (7)
			41	12	33	86

Note: Y = yes; N = no.

Table 5

Simple checklists

Study/tool name/reference ID	Year	Source	Tool purpose	CC	Coh	CS	Items (n)	Development described	Future use	Participants	Variables measure	Other biases	Control confounding	Other statistics	Conflict of interest
Avis15	1994	ED	CA	Y	Y		24	Y	Y	5	N	2	1	1	N
Briggs16	@	W	AMB	Y	Y	Y	5	N	N	1	1	N	1	1	N
Cameron17	2000	ED	SU	Y	Y		36	N	Y	4	3	6	1	2	N
Carneiro18	2002	ED	CA		Y		8	N	Y	1	1	1	1	1	N
CASP CC19	@	W	CA	Y			7	N	Y	2	1	1	1	1	N
CASP Co19	@	W	CA		Y		8	N	Y	1	2	2	2	1	N
CenOccHealth20	@	W	CA	Y	Y	Y	23	N	Y	8	10	2	N	2	Y
CEBM Prog21	@	W	CA		Y		7	Y	Y	1	N	2	1	1	N
CEBM Diag21	@	W	CA	Y	Y	Y	3	Y	Y	1	1	1	N	N	N
DuRantCC22	1994	ED	CA	Y			22	N	Y	6	4	5	3	3	N
DuRantCoh22	1994	ED	CA		Y		24	N	Y	7	8	2	2	3	N
DuRantCS22	1994	ED	CA			Y	18	N	Y	6	4	2	2	3	N
Elwood23	2002	ED	CA	Y	Y		20	Y	Y	Y	2	1	1	1	N
Esdaile24	1985	ED	SU	Y	Y		6	Y	N	N	2	1	1	N	N
Gardner25	1986	ED	AMB	Y	Y		12	N	Y	1	N	N	N	7	N
Hadorn26	1996	ED	AMB		Y		24	Y	Y	8	3	2	1	8	N
HEB Wales27	@	W	CA	Y	Y	Y	13	Y	Y	2	1	4	3	1	N
Horwitz28	1979	ED	CA	Y			12	N	Y	6	2	2	N	N	N
Khan29	@	W	SR	Y			9	N	Y	2	2	2	1	2	N
Khan29	@	W	SR		Y		10	Y	Y	2	1	4	2	1	N
Kilgore30	1981	ED	CA	Y	Y		2	Y	Y	N	N	N	N	N	N
Levine31	1994	ED	CA	Y	Y		7	N	Y	1	1	1	1	2	N
Lichtenstein32	1987	ED	CA	Y			20	Y	Y	4	2	4	2	3	N
London33	@	W	CA	Y	Y		30	Y	Y	4	10	5	5	3	N
Margetts34	2002	ED	SR	Y	Y		6	N	Y	2	2	1	N	2	N
Montreal35	@	W	CA	Y	Y		8	N	Y	2	1	1	1	1	N
Mulrow36	1986	ED	SU		Y		9	Y	N	2	2	2	1	1	N
Newc-Ott CC37	@	W	SR	Y			8	N	Y	4	2	1	1	N	N
Newc-Ott Co37	@	W	SR		Y		8	N	Y	2	3	2	1	N	N
QUADAS38	2003	ED	SR	Y	Y		14	Y	Y	2	3	3	N	N	N
Campbell39	2003	ED	AMB	Y			13	N	Y	Y	Y	N	Y	Y	N
SIGN 50 CC40	@	W	AMB	Y			22	Y	Y	6	1	2	1	3	N
SIGN 50 Co40	@	W	AMB		Y		25	Y	Y	5	3	4	1	3	N
Solomon41	1997	ED	SR	Y	Y		12	N	Y	1	3	1	2	1	N
STARD42	@	W	AMB	Y	Y		14	N	Y	3	4	3	1	2	N
Surgical tutor43	@	W	CA	Y	Y		18	Y	Y	Y	4	2	3	3	N
UCW CC44	@	W	CA	Y			6	Y	Y	1	2	1	2	1	N
UCW Co44	@	W	CA		Y		8	Y	Y	1	N	4	1	1	N
UCW Cross44	@	W	CA			Y	3	Y	Y	1	N	N	1	1	N
Zaza45	2000	ED	SR	Y	Y		15	Y	Y	5	3	3	1	2	N
Zola46	1989	ED	AMB		Y		11	Y	Y	2	2	2	N	2	N

Note: ED, electronic database; W, Internet search; CA, critical appraisal; SR, for conducting systematic reviews; SU, single use in specific context; AMB, ambiguous; these purpose of these tools was not easy to determine and they could be designed for use in guideline development, reporting, critically appraising and/or integrating study data; CC, case-control; Coh, cohort; CS, cross-sectional; NA, not available; NR, not recorded; @, accessed during March 2005; Y, item addressed relevant domain and/or raw score or number of items unavailable; N, domain not addressed.

Table 6

Checklists with an additional summary judgement

Study/tool name/ reference ID	Year	Source	Purpose	CC	Coh	CS	Items (n)	Development described	Future use	Participants	Variables measure	Other biases	Control confounding	Other statistics	Conflict of interest
Bollini74	1992	ED	SU	Y	Y		10	Y	N	3	3	1	2	N	N
Ciliska75	1996	ED	SU	Y	Y		6	Y	N	N	1	1	1	1	N
Cowley76	1995	ED	SU	Y	Y		13	N	N	1	1	3	1	2	1
Effective PH77	@	W	CA	Y	Y		13	N	Y	2	2	2	3	3	N
EPIQ CC78	@	W	CA	Y			30	N	Y	5	8	8	3	3	N
EPIQ Cohort78	@	W	CA		Y		32	N	Y	6	8	10	3	3	N
Fowkes79	1991	ED	CA	Y	Y	Y	22	N	Y	6	3	6	2	2	N
GyorkosCC80	1994	ED	SR	Y			5	Y	Y	2	1	1	1	N	N
GyorkosCoh80	1994	ED	SR		Y		6	Y	Y	1	2	2	1	N	N
GyorkosCS80	1994	ED	SR			Y	4	Y	Y	1	2	N	1	N	N
Spitzer81	1990	ED	SU	Y	Y		17	Y	N	4	4	3	3	2	N
Steinberg82	2000	ED	AMB	Y	Y		24	Y	Y	3	2	5	2	3	N

Note: ED, electronic database; W, Internet search; CA, critical appraisal; SR, for conducting systematic reviews; SU, single use in specific context; AMB, ambiguous; these purpose of these tools was not easy to determine and they could be designed for use in guideline development, reporting, critically appraising and/or integrating study data; CC, case-control; Coh, cohort; CS, cross-sectional; NA, not available; NR, not recorded; @, accessed during March 2005; Y, item addressed relevant domain and/or raw score or number of items unavailable; N, domain not addressed.

Table 7

Scales

Study/tool name/ reference ID	Year	Source	Purpose	CC	Coh	CS	Items (n)	Development described	Future use	Maximum raw score	Participants	Variables measure	Other biases	Control confounding	Other statistics	Conflict of interest
Anders47	1996	ED	SU		Y		6	N	N	6	N	3	2	N	N	N
AriensCC48	2000	ED	SU	Y			18	Y	Y	18	3	9	1	1	2	N
AriensCoh48	2000	ED	SU		Y		17	Y	Y	17	3	7	1	1	2	N
AriensCS48	2000	ED	SU			Y	13	Y	Y	13	2	8	1	1	2	N
Berlin49	1990	ED	SU	Y	Y		16	Y	N	32	N	2	N	2	N	N
Bhutta50	2002	ED	SU	Y			6	N	Y	10	1	N	N	2	N	N
Borghouts51	1998	ED	SU		Y		13	Y	N	13	3	1	1	N	2	N
Campos52	1995	ED	SU	Y	Y		7	N	N	70	N	10	N	N	10	N
Carson53	1994	ED	AMB		Y		10	Y	Y	10	3	N	2	1	2	N
Loney54	@	W	CA	Y	Y	Y	6	N	Y	8	2	2	1	N	1	N
Cho55,b	1994	ED	CA	Y	Y	Y	18	Y	Y	36	12	2	8	4	2	N
Corrao56	1999	ED	SU	Y	Y		16	N	N	30	5	9	4	N	2	N
Downs57	1998	ED	CA	Y	Y		17	Y	Y	21	5	1	3	3	8	N
Garber68	1996	ED	SU	Y	Y	Y	6	N	N	18	N	N	N	N	N	N
Goodman59	1994	ED	AMB	Y	Y		10	Y	Y	50	20	N	5	5	15	N
Jabbour60	1996	ED	SU		Y		7	N	N	7	1	N	1	N	N	N
Kreulen61,c	1998	ED	SU			Y	16	N	N	42	3	12	6	3	12	N
Krogh62	1985	ED	CA	Y	Y	Y	7	N	Y	4	2	N	1	N	1	N
Littenberg63,d	1998	ED	SU	Y	Y	Y	15	N	N	45	NA	NA	NA	NA	NA	N
LongneckerCC64,a	1988	ED	SU	Y			11	N	N	53/58a	(5)	(5)	(5)	(5)	N	N
LongneckerCoh64	1988	ED	SU		Y		4	N	N	20	5	5	5	5	N	N
Macfarlane65	2001	ED	AMB	Y	Y	Y	6	Y	N	6	2	1	2	N	1	N
Manchikanti66	2002	ED	SU	Y	Y		6	Y	N	6	2	2	N	N	1	1
MargettsCC67,a	1995	ED	SR	Y			13	Y	Y	46.4	26.4	10	2	5	5	N
MargettsCoh67	1995	ED	SR		Y		19	Y	Y	53.4	4	22	7	2	9	N
Meijer68	2003	ED	SU		Y		9	Y	Y	9	1	3	1	1	2	N
Nguyen69	1999	ED	SU	Y	Y	Y	14	N	N	72	4	18	6	12	20	N
Rangel70	2003	ED	AMB		Y		15	Y	N	17	3	5	1	N	6	N
Reisch71,a	1989	ED	CA	Y	Y		35 (min)	N	Y	% items fulfilled	(1)	(1)	(1)	N	(1)	N
Stock72	1991	ED	SU	Y	Y	Y	7	N	N	21	6	6	3	3	N	N
WindtCC73	2000	ED	SU	Y			20	Y	N	20	3	11	4	1	4	N
WindtCoh73	2000	ED	SU		Y		18	Y	N	18	3	8	1	1	4	N
WindtCS73	2000	ED	SU			Y	16	Y	N	16	3	8	1	1	4	N

Note: ED, electronic database; W, Internet search; CA, critical appraisal; SR, For conducting systematic reviews; SU, Single use in specific context; AMB, Ambiguous; these purpose of these tools was not easy to determine and they could be designed for use in guideline development, reporting, critically appraising and/or integrating study data; CC, case-control; Coh, cohort; CS, cross-sectional; NA, not available; NR, not recorded; @, accessed during March 2005; Y, item addressed relevant domain and/or raw score or number of items unavailable; N, domain not addressed.

^aThese tools were extremely complex and require considerable input to calculate raw scores and to convert to final scores, depending on the primary study design and methods.

^bThis tool allowed the possibility of different total scores based on study design and applied differential weighting, and included case studies and randomized trials within a single scale.

^cWeighting was applied to the raw scores by a factor of 2 for study methodology, evaluation methodology and by a factor of 1.5 for statistical methodology.

^dThe scale is not described in sufficient detail to assess weighting in domains.

The biggest group was checklists (41; 48%),^15–46 followed by scales (33; 38%)^47–73 and finally summary judgement checklists (12; 14%)^74–82. Fifteen per cent of all tools were for generic use in systematic reviews, one-third for use in critical appraisal, one-third for single use in a specific systematic review and 15% where the purpose was ambiguous. For checklists, half were critical appraisal tools (22; 54%) whilst two-thirds of scales were review-specific (21; 64%). Over half of all tools (54%) described their development process in detail.

Just under three-quarters of all tools were proposed as being suitable for future use, including all of the critical appraisal tools and generic systematic review tools and six of the tools originally designed for use in a specific systematic review.

A number of tools were designed to address specific study design types: case-control studies alone (19%); cohort studies alone (27%) and cross-sectional studies alone (7%) (Table 3). Others addressed different combinations of these design types, with almost one-third addressing both case-control and cohort studies (45%) and 15% addressing all three. The number of items in all tools ranged from 3 to 36, with a mean of 13.7 (13.4 for simple checklists, 15.2 for simple checklists with a summary judgement and 12.6 for scales).

The majority of tools included items relating to methods for selecting study participants (92%). The proportion of tools including items about the measurement of study variables (exposure, outcome and/or confounding variables) was also high (86%). Assessment of other design-specific sources of bias (including recall bias, interviewer bias and biased loss to follow-up but excluding confounding) was included in 86%, around three-quarters assessed control of confounding (78%) and three-quarters included items concerning statistical methods (78%). Conflict of interest was included in only three tools (3%).

To address weighting, we recorded the number of items included in both types of checklists devoted to each of our key domains, whilst for scales we recorded the total available raw score for each domain. As can be seen from Tables 5 to 7, there is a little consistency among tools, with considerable variability in the number of items across domains and across tool types.

Discussion

Assessing the quality of evidence from observational epidemiological studies requires tools that are designed and developed with this specific purpose in mind. To our knowledge, this is the most comprehensive search to date of both the medical literature and the Internet for tools to assess such studies. We have identified 86 candidate tools, comprising checklists, summary judgement checklists and scales. The Internet search identified three more tools that were not identified through searching electronic databases. Future search strategies may wish to employ similar methodologies to ensure the identification of all available tools, articles or studies. Despite the comprehensive nature of the search strategy employed, it is unlikely that all existing tools for assessing quality of observational epidemiological studies have been identified, since many are developed for specific systematic reviews, and it is very difficult to identify all of these through searching electronic databases.

A large number of the tools were scales that resulted in numerical summary scores. Whilst this approach has the appearance of simplicity, considerable concerns have been raised about such an approach to assessing quality.⁸³ Summary scores involve inherent weighting of component items, some of which may not be directly related to the validity of a study's findings (such as sample size calculations). It is unclear how weights for different items should be determined, and different scales may reach different conclusions on the overall quality of an individual study.⁸⁴ We have found that the weighting applied in scales to different study domains is variable and inconsistent. Similar considerations apply to summary judgement checklists, although qualitative rather than quantitative summaries may be less prone to inappropriate analysis. We prefer a more transparent checklist approach that concentrates on the few, principal, potential sources of bias in a study's findings.

Tool components should, where possible, be based on empirical evidence of bias, although this may be difficult to obtain, and there is a need for more empirical research on relationships between specific quality items and findings from epidemiological studies. There was wide variation among tools in the number and nature of items, scoring ranges (where applicable) and levels of development. The specific components assessed by the tools differed across both study design and tool type. Although we have not implemented all tools, we would anticipate that different tools would indicate different degrees of quality when applied to the same study.

It is encouraging that most tools included items to assess methods for selecting study participants (92%) and to assess methods for measuring study variable and design-specific sources of bias (both 86%). Over three-quarters of tools assessed the appropriate use of statistics, and the control of confounding (both 78%) but conflict of interest was only included in 4% of tools. Around one-third of the tools were designed for specific clinical or research topics, limiting their wider applicability; there was a marked difference between tool types in this respect, with the majority of checklists designed for critical appraisal and the majority of scales for single use in specific single reviews. The ambiguity of purpose of some of the tools is a cause for concern, and more clarity is needed to differentiate assessments of the quality of reporting from the quality of what was actually done in the study.

A rigorous development process should be an important component of tool design, but only half of the tools provided a clear description of their design, development or the empirical basis for item inclusion or evaluation of the tool's validity and reliability. This is of particular concern as 70% of the tools were proposed as being suitable for future use in other contexts. Future tools should undergo a rigorous development process to ensure that they are evidence-based, easy to use and readily interpretable.

This review has highlighted the lack of a single obvious candidate tool for assessing quality of observational epidemiological studies. One might regard this review as the first stage towards development of a generic tool. In such an endeavour, one would need to reach a consensus on the critical domains that should be included. The development of the STROBE statement has involved extensive discussion among numerous experienced epidemiologists and statisticians. Despite targeting the reporting of studies, many items were no doubt selected due to presumed (or evidence of) association with susceptibility to bias. Thus the statement should provide a suitable starting point for development of a quality assessment tool, and we have been guided by it in our presentation of results.

Around half of the checklists included what we regard as the three most fundamental domains of appropriate selection of participants, appropriate measurement of variables and appropriate control of confounding; all were considered appropriate for future use. The majority of these tools also included items on potential design-specific biases. However, we are reluctant to recommend a specific tool, without having implemented them all on multiple studies with a view to assessing their properties and ease-of-use. Our broad recommendations are that tools should (i) include a small number of key domains; (ii) be as specific as possible (with due consideration of the particular study design and topic area); (iii) be a simple checklist rather than a scale and (iv) show evidence of careful development, and of their validity and reliability.

Search strategy

(1 or 2 or 3 or 4) AND (5 or 6 or 7) AND (8 or … to 17)

• scale*

• checklist*

• critical apprais*

• tool*

• valid*

• quality

• (bias* OR confounding) AND (assess* OR measure* OR evaluat*)

• OBSERVATIONAL STUDIES (MeSH)

• observational stud*

• COHORT STUDIES (MeSH)

• cohort stud*

• CASE-CONTROL STUDIES (MeSH)

• case-control stud*

• CROSS-SECTIONAL STUDIES (MeSH)

• cross-sectional stud*

• FOLLOW-UP STUDIES (MeSH)

• follow-up stud*

Conflict of interest: None declared.

KEY MESSAGES

• Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies.

• Only about half of the identified tools did not describe their development or validity and reliability.

• Tools for assessing quality should be rigorously developed, evidence-based, valid, reliable and easy to use and concentrate on assessing sources of bias.

• There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.