Abstract

Background Forest plots are graphical displays of findings of systematic reviews and meta-analyses. Little is known about the style and content of these plots and whether published plots maximize the graphic’s potential for information exchange.

Methods We examine the number, style and content of forest plots presented in a previously studied cross-sectional sample of 300 systematic reviews. We studied all forest plots in non-Cochrane reviews and a sample of forest plots in Cochrane reviews.

Results The database contained 129 Cochrane reviews and 171 non-Cochrane reviews. All the Cochrane reviews had forest plots (2197 in total), and a random sample of 500 of these plots were included. In total, 28 of the non-Cochrane reviews had forest plots (139 in total), all of which were included. Plots in Cochrane reviews were standardized but often contained little data (80% had three or fewer studies; 10% had no studies) and always presented studies in alphabetical order. Non-Cochrane plots depicted a larger number of studies (60% had four or more studies) and 59% ordered studies by a potentially meaningful characteristic, but important information was often missing. Of the 28 reviews that had a forest plots with at least 10 studies, 3 (11%) had funnel plots.

Conclusions Forest plots in Cochrane reviews were highly standardized but some of the standards do not optimize information exchange, and many of the plots had too little data to be useful. Forest plots in non-Cochrane reviews often omitted key elements but had more data and were often more thoughtfully constructed.

Introduction

Systematic reviews are an important means of summarizing the methods and results of individual studies and increasingly are being used as a starting point in the development of clinical practice guidelines1 and have been advocated as the starting and ending point of all randomized trials.2 Forest plots—the graphical display of individual study results and, usually, the weighted average of studies included in a systematic review—are one way of summarizing the review’s results for a specific outcome.3–6 Plots of this kind first appeared in the 1970s and were refined over the next two decades; they were first called ‘forest plots’ in the mid-1990s.4 Since that time the elements contained in a forest plot and the layout of such plots have become somewhat standardized, largely due to the introduction of software that helps authors construct these plots (Figure 1).7,8

A standardized format for forest plots no doubt helps readers because repeated exposure to a familiar format decreases the time and effort required to become oriented to the graphic and likely facilities their interpretation. Nevertheless, many of the de facto standards for the construction of forest plots were not based on theory or empirical evidence regarding optimal information exchange and, for a number of issues, theory would suggest that current practice is suboptimal.

Figure 1

The plot is drawn in STATA 11 (Stata Corp., College Station, TX, USA) from data presented in Ezekowitz et al.22 Note that studies have been sorted first by whether they addressed primary or secondary prevention and second by year of publication. This organization allows readers to easily determine whether they believe that these variables affect the outcome.

Figure 1

The plot is drawn in STATA 11 (Stata Corp., College Station, TX, USA) from data presented in Ezekowitz et al.22 Note that studies have been sorted first by whether they addressed primary or secondary prevention and second by year of publication. This organization allows readers to easily determine whether they believe that these variables affect the outcome.

Recently, Moher et al. characterized many of the qualities of the text and tables in a group of systematic reviews.9 We now examine the forest plots contained in this set of systematic reviews with the goal of describing current practice regarding their construction and display. By defining current practice we hope to identify ways that forest plots in future systematic reviews can be improved.

Methods

We used the database assembled by Moher et al., which consists 300 English language reports of systematic reviews (of various study designs) that were indexed on Medline during November 2004.9 These papers were found by reviewing 1046 potential citations and keeping those that were systematic reviews, which offered explicit methods for article identification and eligibility. The 300 papers were found in 132 journals, and mainly reviewed therapeutic (71%), epidemiologic (13%) and diagnostic (8%) questions. A total of 54% of the systematic reviews did some type of mathematical pooling of individual study results.

We reviewed each paper for the presence of any graphical attempt to simultaneously portray the results of the individual studies included in the systematic review. We also noted whether each paper contained funnel plots—a graphical display used to assess asymmetry of results, a possible explanation for publication bias.10 We did not count isolated data tables; we required that there be some form of graphic presentation of the individual study data.

We counted the number of plots in each paper and noted whether each paper was a Cochrane review.11 All plots from non-Cochrane reviews were included. We sampled the plots in Cochrane reviews because the number of forest plots they contained substantially outnumbered the forest plots found in non-Cochrane reviews. We used the random number function in STATA 9.0 to select two plots from every Cochrane review. We completed the sampling by randomly selecting from the pool of remaining Cochrane review plots sufficient plots to bring the total sample of Cochrane review plots to 500.

Plots were independently rated by two members of the research team (J.V. and T.H.) who had been trained by the principal investigator (D.S.) and had proved their accuracy on a set of training plots. Data forms were reviewed for inconsistencies between the raters, and the principal investigator adjudicated discrepancies. Forest plots were assessed for: whether the individual studies were separated into sub-panels and on what basis; how the studies in each plot (or each sub-panel if there were sub-panels) were ordered—alphabetically, by effect size, by weight, by year of publication, by study characteristic (e.g. dose used); and what measure of effect was used, what scale was used and what data elements were presented in the graphic.

Results

Of the 300 systematic reviews in the data-set, 129 (43%) were Cochrane reviews (Table 1). Although all the reviews had forest plot frameworks, only 115 (89%) had data in the frameworks (in 14 Cochrane reviews all forest plot frameworks were empty as no eligible studies were included). There were 2197 individual forest plots (although ∼10% had no data, see below), a mean of 17 plots per Cochrane review [median 9, interquartile range (IQR) 4–24]. The maximum number of forest plots in a Cochrane review was 125.

Table 1

Number of forest and funnel plots in 300 systematic reviews

Number Cochrane Non- Cochrane Total 
Papers 129 171 300 
Papers with forest plots (%) 115a (89) 28 (16) 143 (48) 
Forest plots 2197 139 2336 
Plots per paper, median (IQR) 9 (4, 24) 2 (1, 4) 7 (3, 16) 
Papers with funnel plots (%) 4 (3) 5 (3) 9 (3) 
Number Cochrane Non- Cochrane Total 
Papers 129 171 300 
Papers with forest plots (%) 115a (89) 28 (16) 143 (48) 
Forest plots 2197 139 2336 
Plots per paper, median (IQR) 9 (4, 24) 2 (1, 4) 7 (3, 16) 
Papers with funnel plots (%) 4 (3) 5 (3) 9 (3) 

aAll 129 Cochrane papers had forest plot frameworks but 14 had no studies in the framework.

Of the 171 non-Cochrane reviews, 28 (16%) had at least one forest plot. Non-Cochrane reviews with a forest plot had a mean of five plots (median 2, IQR 1–4). The three studies with the largest number of plots had 44, 20 and 9. A total of 3% (4/129) of Cochrane and 3% (5/171) of non-Cochrane reviews had funnel plots. Of the 28 reviews that had a forest plot that contained at least 10 studies, 3 (11%) had funnel plots.

The number of individual studies represented within the forest plots is presented in Table 2. In general, there were fewer studies in the plots presented in Cochrane reviews than in non-Cochrane reviews. This was true for plots that contained a single panel of studies (median one vs seven studies) and plots that divided the studies into sub-panels (median one vs two studies per sub-panel). One paper12 contributed 44 (32%) of 139 non-Cochrane plots. These plots were atypical of the other non-Cochrane plots—88 of the 92 panels and sub-panels had one study, 3 had two studies and 1 had three studies. Excluding this paper, single panel non-Cochrane plots had a median of eight studies and multi-paneled plots had a median five studies per panel. Although 36% of non-Cochrane plots that used sub-panels (71% if we exclude the one aberrant study) had two sub-panels that contained at least four studies, only 16% of Cochrane plots did so. When present, sub-panels were organized by an explanatory variable in 78% of Cochrane and 70% of non-Cochrane studies with the remaining plots organized by outcome variables. (Table S1, Supplementary data are available at IJE online).

Table 2

Number of studies in forest plots

  Non-Cochrane
 
 Cochrane All Typicala 
Number of plots 500b 139 95 
Plots with a single panel of studies n (%) 297 (59) 80 (58) 67 (71) 
        0 29 (10) 0 (0) 0 (0) 
        1–2 196 (66) 18 (23) 6 (9) 
        3–4 34 (12) 10 (12) 9 (13) 
        >4 36 (12) 52 (65) 52 (78) 
    Studies per plot, mean 11 
    Median (IQR) (range) 1 (1–3) (1–19) 7 (3–11) (1–40) 8 (5–12) (1–40) 
Plots with sub-panels 203 (41) 59 (42) 28 (29) 
    Number of sub-panels 749 171 92 
    Sub-panels per plot (mean) – 
    Median (IQR) (range) 2 (2–3) (2–33) 3 (2–3) (2–8) – 
Sub-panels with no. of studies, n (%)    
        0 92 (12) 0 (0) 0 (0) 
        1–2 489 (65) 96 (56) 17 (18) 
        3–4 79 (11) 25 (15) 25 (27) 
        >4 89 (12) 50 (29) 50 (55) 
    Studies per sub-panel, mean 
    Median (IQR) (range) 1 (1–2) (0–38) 2 (1–5) (1–22) 5 (3–9) (1–22) 
    Articles with at least one sub-panel with no. of studies, n (%)    
        >2 83 (41) 28 (47) 28 (100) 
        >3 61 (30) 26 (44) 26 (93) 
    Articles with at least two sub-panels with no. of studies, n (%)    
        >2 53 (26) 21 (36) 21 (75) 
        >3 33 (16) 20 (36) 20 (71) 
  Non-Cochrane
 
 Cochrane All Typicala 
Number of plots 500b 139 95 
Plots with a single panel of studies n (%) 297 (59) 80 (58) 67 (71) 
        0 29 (10) 0 (0) 0 (0) 
        1–2 196 (66) 18 (23) 6 (9) 
        3–4 34 (12) 10 (12) 9 (13) 
        >4 36 (12) 52 (65) 52 (78) 
    Studies per plot, mean 11 
    Median (IQR) (range) 1 (1–3) (1–19) 7 (3–11) (1–40) 8 (5–12) (1–40) 
Plots with sub-panels 203 (41) 59 (42) 28 (29) 
    Number of sub-panels 749 171 92 
    Sub-panels per plot (mean) – 
    Median (IQR) (range) 2 (2–3) (2–33) 3 (2–3) (2–8) – 
Sub-panels with no. of studies, n (%)    
        0 92 (12) 0 (0) 0 (0) 
        1–2 489 (65) 96 (56) 17 (18) 
        3–4 79 (11) 25 (15) 25 (27) 
        >4 89 (12) 50 (29) 50 (55) 
    Studies per sub-panel, mean 
    Median (IQR) (range) 1 (1–2) (0–38) 2 (1–5) (1–22) 5 (3–9) (1–22) 
    Articles with at least one sub-panel with no. of studies, n (%)    
        >2 83 (41) 28 (47) 28 (100) 
        >3 61 (30) 26 (44) 26 (93) 
    Articles with at least two sub-panels with no. of studies, n (%)    
        >2 53 (26) 21 (36) 21 (75) 
        >3 33 (16) 20 (36) 20 (71) 

aThis column reports non-Cochrane results with one large, atypical paper12 removed. See text.

bWe randomly sampled 500 plots from the 2137 Cochrane plots.

All Cochrane review plots displayed individual study results in alphabetical order, either by first author last name or study acronym (Table 3). In contrast, 46% of non-Cochrane review plots displayed study results by year of publication and a smaller number sorted the study results by effect size, and sample size. Ratio measures were the predominant reported outcome and most were scaled between either 0.01 and 100 or 0.1 and 10. Cochrane review plots were always scaled symmetrically whereas non-Cochrane review plots used a variety of scales. A logarithmic scale was used for all plots of ratio measures. In all Cochrane plots the symbol used to indicate the estimated effect size (e.g. mean, relative risk) for each study was sized to reflect that study’s weight. A total of 17 of 28 (61%) non-Cochrane papers (49% of all plots) had sized symbols.

Table 3

Characteristics of forest plots

 Cochrane Non- Cochrane 
Ordering of studies (>1 study in plot or a sub-panel) n = 230 n = 99 
    Year of publication, n (%) 0 (0) 45 (46) 
    Alphabetical by author or  acronym 230 (100) 24 (24) 
    Effect size 0 (0) 9 (9) 
    Number of participants 0 (0) 1 (1) 
    Method not apparent 0 (0) 20 (20) 
Reported statistic (Plots with at least 1 study) n = 468 n = 139 
    Ratio measure (risk, odds,  hazard) 282 (60) 61 (44)a 
    Absolute risk difference  (categorical data) 5 (1) 2 (2) 
    Difference (continuous measure) 181 (39) 73 (52) 
    Cannot be determined from  figure 0 (0) 3 (2) 
Scale (for plots that used ratio measures) n = 282 n = 61 
    0.001–1000 25 (9) 1 (2) 
    0.01–100 55 (20) 27 (44) 
    0.1–10 194 (69) 19 (31) 
    0.2–5 7 (2) 0 (0) 
    0.5–2 1 (0) 2 (3) 
    Other 0 (0) 12 (20) 
 Cochrane Non- Cochrane 
Ordering of studies (>1 study in plot or a sub-panel) n = 230 n = 99 
    Year of publication, n (%) 0 (0) 45 (46) 
    Alphabetical by author or  acronym 230 (100) 24 (24) 
    Effect size 0 (0) 9 (9) 
    Number of participants 0 (0) 1 (1) 
    Method not apparent 0 (0) 20 (20) 
Reported statistic (Plots with at least 1 study) n = 468 n = 139 
    Ratio measure (risk, odds,  hazard) 282 (60) 61 (44)a 
    Absolute risk difference  (categorical data) 5 (1) 2 (2) 
    Difference (continuous measure) 181 (39) 73 (52) 
    Cannot be determined from  figure 0 (0) 3 (2) 
Scale (for plots that used ratio measures) n = 282 n = 61 
    0.001–1000 25 (9) 1 (2) 
    0.01–100 55 (20) 27 (44) 
    0.1–10 194 (69) 19 (31) 
    0.2–5 7 (2) 0 (0) 
    0.5–2 1 (0) 2 (3) 
    Other 0 (0) 12 (20) 

aWhen the one atypical article12 with 31 plots is removed, ratio measures account for 56% and difference measures for 40% of the remaining 108 studies.

The typical Cochrane review plot includes: (i) a title that states the research question, the comparison being made and the outcome measure; (ii) a description of each study including author last name, publication year, the n/N (binary outcome) or N (continuous outcome) for each group, the point estimate and 95% CI both numerically and as a graphic, and the weight that the study was given if meta-analysis was performed; (iii) for each meta-analysis—a summary diamond, a pooled estimate of the outcome and its CI, the total N for binary measures, a test of heterogeneity and a test for overall effect; and (iv) a scale for the forest plot with labels indicating which direction favours one group or the other (Table 4 and Figure 1). With the exception of one plot that did not indicate which direction favoured the treatment group, all Cochrane review plots contained all these elements except those listed in (iii) above when meta-analysis was not performed. The non-Cochrane review plots were less standardized in this regard, with roughly half of the plots missing many of the elements outlined above (Table 4). In particular, the majority did not include the summary results of each of the studies depicted in the plot.

Table 4

Items reported in forest plots

 Cochrane Non- Cochrane 
Items about individual studies n = 500 n = 139 
    Study identity, n (%) 500 (100) 134 (96) 
    Number of participants in  each group 500 (100) 44 (32) 
    Summary data; e.g. n/N,  mean (SD) 500 (100) 39 (28) 
    Effect size and CI 500 (100) 75 (54) 
Items concerning layout of plot   
    Labels indicating which  group is favoured 499 (100) 112 (81) 
Items about meta-analysis results n = 221 n = 96 
    Weight given to each study 221 (100) 46 (48) 
    Test for overall effect 221 (100) 38 (40) 
    Test for heterogeneity 221 (100) 50 (52) 
 Cochrane Non- Cochrane 
Items about individual studies n = 500 n = 139 
    Study identity, n (%) 500 (100) 134 (96) 
    Number of participants in  each group 500 (100) 44 (32) 
    Summary data; e.g. n/N,  mean (SD) 500 (100) 39 (28) 
    Effect size and CI 500 (100) 75 (54) 
Items concerning layout of plot   
    Labels indicating which  group is favoured 499 (100) 112 (81) 
Items about meta-analysis results n = 221 n = 96 
    Weight given to each study 221 (100) 46 (48) 
    Test for overall effect 221 (100) 38 (40) 
    Test for heterogeneity 221 (100) 50 (52) 

SD, Standard deviation.

The majority of Cochrane review plots presented summary diamonds—graphical representations of the summation of the findings of the individual studies derived from meta-analytic techniques (Table S1, Supplementary data are available at IJE online). Summary diamonds were presented in 70% of plots that had only one study and were used more judiciously in non-Cochrane review plots where they were seldom provided unless there were several studies to be combined. Of the 28 non-Cochrane papers with forest plots, 12 (43%) did not state the statistical method by which the summary diamond was created. Of the other 16 articles, 8 (50%) stated the method used random effects, 5 (31%) used fixed effects, 1 (6%) had some plots that used fixed and others random effects and 1 (6%) reported two summary diamonds, one for each method. In the 115 Cochrane reviews 64% used fixed effects, 17% random effects, 15% had some plots for which each method was used and 4% used neither. The ratio of fixed effects to random effects meta-analytic techniques was roughly 4:1 in Cochrane review plots and 3:4 in non-Cochrane review plots.

Plots in all Cochrane and 8 (28%) of the 28 non-Cochrane reviews that had plots appear to have been generated in Review Manager (RevMan), available at the Cochrane website (http://www.cc-ims.net/RevMan). Only 8 of the 28 non-Cochrane reviews stated the software used to create the plots (6 RevMan, 2 Stats Direct), although we can assume, based on forest plot style and the software used for the statistical analysis, that some authors used Stata and StatXact. In 18 of 28 papers it was not clear to us the software that was used to make the plot, and in 11 of 28 papers the authors neither stated the software used to make the plot nor the software used to perform the analyses.

Discussion

Forest plots are a concise graphical way of summarizing the quantitative findings of a systematic review. Such plots are informative whether they contain a summary diamond from a meta-analysis of the included study results or just present the results of individual studies. Our cross-sectional study reveals several important findings. First, authors of Cochrane reviews generally follow a recipe whereby forest plots are created based on the existence of a question rather than the availability of data. As a result, all Cochrane reviews had forest plots, but 10% contained no data and >65% contained just one or two studies. While these sparsely populated plots certainly emphasize that “more research is needed,” plots with 0 or 1 studies serve no other purpose and the message that data are sparse could be made more efficiently. Of note, the 2008 version of the ‘Cochrane Handbook for Systematic Reviews of Interventions’ states: ‘Forest plots should not be generated that contain no studies, and are discouraged when only a single study is found for a particular outcome’.13 The paucity of information contained in forest plots with sparse data is exacerbated when these plots present summary diamonds that ‘summarize’ one study. The information in these diamonds is redundant and may falsely inflate readers’ assessments of the amount of available information.

The majority (84%) of non-Cochrane reviews did not contain even one forest plot. Although it is possible that the authors of some reviews had ample data from many studies but naively did not know to include such a plot, there was little evidence of this. More commonly, when data were sparse and the plot would have contained less than three studies, authors of non-Cochrane reviews wisely decided to omit the plot. As a result, when non-Cochrane reviews did have plots these plots tended to be richer; they contained sufficient papers to make the plot interesting and helpful.

Only 3% of both Cochrane and non-Cochrane reviews presented funnel plots. Some authors believe that when there are a sufficient number of studies (e.g. 10 or more), funnel plots can be very useful for detecting asymmetry, perhaps suggesting publication bias,10,14 whereas others argue that funnel plots are not particularly helpful.15,16 Regardless, our study suggests that they are reported infrequently.

Scientists use graphics to communicate because graphics can have far greater data density and multidimensionality than text.17 Although the horizontal dimension of each forest plot represents the magnitude and precision of each study’s result, our data demonstrate that authors’ use of forest plots does not exploit the vertical dimension. All Cochrane review plots and 44% of non-Cochrane review plots presented studies in either alphabetical order or some other order that had little potential for illuminating the meaning of the data. This is unfortunate as one of the main benefits of a systematic review is the opportunity to explore why study results differ from one another.18 For example, when ordered by year of publication, forest plots can reveal trends related to changing technologies (early studies of computerized tomography will have lower sensitivity than more recent ones because scan resolution has improved). They can also be useful when cumulative meta-analysis is performed or to show how beliefs are modified by the addition of new data to an existing meta-analysis (i.e. updating systematic reviews).2,19 Ordering by effect size can aid in the detection of heterogeneity, and ordering on sample size or some analogue (e.g. study weight) can aid in the detection of publication bias (in a manner similar to funnel plots). When ordered by a characteristic of the studies (e.g. dosage used, or severity of illness of the subjects, risk of bias), plots may reveal patterns that would otherwise go unobserved (Figure 2). It is therefore unfortunate that alphabetical order, which wastes the vertical dimension, predominates. We found no directions regarding this concern in the 2008 Cochrane Handbook.13

Figure 2

Effect of ordering on appearance of forest plot. (A) Ordered by author last name. (B) Ordered by dose. These data, adapted from Annane et al.23, are shown ordered by author last name and the dose employed by each study. Within each dosage level, studies are ordered by effect size. From the bottom panel we easily see that there is a suggestion that dose is an effect modifier. This is not apparent in the upper panel.

Figure 2

Effect of ordering on appearance of forest plot. (A) Ordered by author last name. (B) Ordered by dose. These data, adapted from Annane et al.23, are shown ordered by author last name and the dose employed by each study. Within each dosage level, studies are ordered by effect size. From the bottom panel we easily see that there is a suggestion that dose is an effect modifier. This is not apparent in the upper panel.

All Cochrane review forest plots were created using RevMan software, which ensured that they appeared in a standard format. The benefit of this strategy is homogeneous-appearing forest plots that contain all the desired elements (Table 4). A downside of this approach is that these elements appear even when they are wholly irrelevant. In contrast, authors of non-Cochrane reviews employed a variety of methods to create their plots. As a result, their plots are not standardized and many omit important information (Table 4). However, these plots seldom display nonsense graphics (e.g. meta-analyses of single studies).

Our findings provide a baseline from which authors, peer reviewers and editors can contemplate how to further improve the information content and organization of forest plots. The most obvious first steps are listed in Figure 3. We emphasize that authors should: (i) only use forest plots when there are sufficient studies to make them of value; (ii) ensure that plots contain all the important elements; and (iii) exploit the plot’s vertical dimension by ordering studies in a way that might illustrate important differences among them, such as by year of publication, effect size or important study characteristic.

Figure 3

Suggestions for making high-quality forest plots. aMedian (IQR) may be more appropriate when data are skewed though there is no widely used mechanism for pooling data in this form.

Figure 3

Suggestions for making high-quality forest plots. aMedian (IQR) may be more appropriate when data are skewed though there is no widely used mechanism for pooling data in this form.

We also encourage authors to consult item 21 of the PRISMA Statement20 [‘For all outcomes considered (benefits or harms) present, for each study: (i) simple summary data for each intervention group, (ii) effect estimates and confidence intervals, ideally with a forest plot’], and the accompanying PRISMA explanation document.21 That paper provides examples of ‘good reporting’ including use of tables and graphics to present the results of systematic reviews, along with an explanation and evidence, when available, for reporting this information.

Supplementary Data

Supplementary data are available at IJE online.

Funding

This work was supported in part by an unrestricted grant from the Korein Foundation to D.L.S. Cancer Research UK to D.G.A. and University of Ottawa Research Chair to D.M.

Acknowledgements

The authors thank Jennifer Tetzlaff for her help with file management, and Jessica Brown, Louis Muscarella and Gina Pang for their help in conducting this research.

Conflict of interest: None declared.

KEY MESSAGES

  • Plots in Cochrane reviews were standardized but often contained little data (80% had 3 or fewer studies; 10% had no studies) and always presented studies in alphabetical order.

  • Non-Cochrane plots depicted a larger number of studies (60% had 4 or more studies) and 59% ordered studies by a potentially meaningful characteristic, but important information was often missing.

  • We emphasize that authors should: 1) only use forest plots when there are sufficient studies to make them of value 2) ensure that plots contain all of the important elements, and 3) exploit the plot's vertical dimension by ordering studies in a way that might illustrate important differences among them, such as by year of publication, effect size, or important study characteristic.

References

1
Egger
M
Davey Smith
G
Altman
DG
Systematic Reviews in Health Care: Meta-Analysis in Context
 , 
2001
2nd
London, England
BMJ Books
2
Clarke
M
Hopewell
S
Chalmers
I
Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report
J R Soc Med
 , 
2007
, vol. 
100
 (pg. 
187
-
90
)
3
Ried
K
Interpreting and understanding meta-analysis graphs—a practical guide
Aust Fam Physician
 , 
2006
, vol. 
35
 (pg. 
635
-
38
)
4
Lewis
S
Clarke
M
Forest plots: trying to see the wood and the trees
BMJ
 , 
2001
, vol. 
322
 (pg. 
1479
-
80
)
5
Hyde
CJ
Stanworth
SJ
Murphy
MF
Can you see the wood for the trees? Making sense of forest plots in systematic reviews
Transfusion
 , 
2008
, vol. 
48
 (pg. 
218
-
20
)
6
Perera
R
Heneghan
C
Interpreting meta-analysis in systematic reviews
Evid. Based Med
 , 
2008
, vol. 
13
 (pg. 
67
-
69
doi:10.1136/ebm.13.3.67
7
Revman
(13 October 2008, date last accessed) 
8
STATA 10. Stata Corp
College Station TX
(13 October 2008, date last accessed) 
9
Moher
D
Tetzlaff
J
Tricco
A
Sampson
M
Altman
DG
Epidemiology and reporting characteristics of systematic reviews
PLoS Med
 , 
2007
, vol. 
4
 pg. 
e78
 
10
Egger
M
Davey Smith
G
Schneider
M
Minder
C
Bias in meta-analysis detected by a simple, graphical test
BMJ
 , 
1997
, vol. 
315
 (pg. 
629
-
34
)
11
Cochrane Collaboration website
(10 October 2008, date last accessed) 
12
Ottawa
Panel
Evidence-based clinical practice guidelines for therapeutic exercises in the management of rheumatoid arthritis in adults
Phys Ther
 , 
2004
, vol. 
84
 (pg. 
934
-
72
)
13
Higgins
JPT
Green
S
Cochrane Handbook for Systematic Reviews of Interventions
  
Version 5.0.0 (updated February 2008). The Cochrane Collaboration, 2008, Section 11.3.2.1. http://www.cochrane-handbook.org/ (10 October 2008, date last accessed)
14
Sterne
JA
Egger
M
Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis
J Clin Epidemiol
 , 
2001
, vol. 
54
 (pg. 
1046
-
55
)
15
Ioannidis
JP
Trikalinos
TA
The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey
CMAJ
 , 
2007
, vol. 
176
 (pg. 
1091
-
96
)
16
Lau
J
Ioannidis
JP
Terrin
N
Schmid
CH
Olkin
I
The case of the misleading funnel plot
BMJ
 , 
2006
, vol. 
333
 (pg. 
597
-
600
)
17
Tufte
ER
The Visual Display of Quantitative Information
 , 
1983
Cheshire, CT
Graphics Press
18
Greenland
S
O’Rourke
K
Rothman
KJ
Greenland
S
Lash
TL
Meta-analysis
Modern Epidemiology
 , 
2008
3rd
Philadelphia, PA
Lippincott, Williams and Wilkins
(pg. 
652
-
82
)
19
Lau
J
Antman
EM
Jimenez-Silva
J
Kupelnick
B
Mosteller
F
Chalmers
TC
Cumulative meta-analysis of therapeutic trials for myocardial infarction
N Engl J Med
 , 
1992
, vol. 
327
 (pg. 
248
-
54
)
20
Moher
D
Liberati
A
Tetzlaff
J
Altman
DG
the PRISMA Group
Preferred Reporting items for systematic reviews and meta-analyses
The PRISMA Statement Ann Intern Med
 , 
2009
, vol. 
151
 (pg. 
264
-
69
)
21
Liberati
A
Altman
DG
Tetzlaff
J
, et al.  . 
and the PRISMA Group. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration
PLoS Med
 , 
2009
, vol. 
6
 pg. 
e1000100
 
22
Ezekowitz
JA
Rowe
BH
Dryden
DM
, et al.  . 
Systematic review: implantable cardioverter defibrillators for adults with left ventricular systolic dysfunction
Ann Intern Med
 , 
2007
, vol. 
147
 (pg. 
251
-
62
)
23
Annane
D
Bellissant
E
Bollaert
PE
Briegel
J
Keh
D
Kupfer
Y
Corticosteroids for severe sepsis and septic shock: a systematicreview and meta-analysis
BMJ
 , 
2004
, vol. 
329
 (pg. 
480
-
4
)