Quality of reporting of systematic reviews and meta-analyses of surgical randomized clinical trials.

Background Well designed and conducted systematic reviews are essential to clinical practice. Surgical intervention is more complex than medical intervention when considering special items related to procedures. There has been no cross‐sectional study of the reporting quality of systematic reviews of surgical randomized trials focused on special items relating to surgical interventions. Methods A cross‐sectional survey of systematic reviews of surgical randomized trials published in 2007 and 2017 was undertaken via a PubMed search. Quality of reporting was assessed by the PRISMA checklist, with intervention details containing 27 items. Univariable and multivariable linear regression was used to explore factors in the checklist as indicators of reporting quality. Results A total of 204 systematic reviews were identified. The median score for the PRISMA checklist was 22 (i.q.r. 20–24), and systematic reviews published in 2017 had a significantly higher median score than those from 2007 (22 (i.q.r. 21–24) versus 20 (17–22); P < 0·001). Among the 27 items, 15 were reported adequately and three were reported poorly (in less than 50 per cent of reports). The proportion of other items reported ranged from 54·4 to 77·9 per cent. In multivariable analysis, systematic reviews published in 2017 (coefficient 0·59, 95 per cent c.i. 0·50 to 0·69) and Cochrane reviews (coefficient 0·67, 0·55 to 0·81) were associated with better reporting. Conclusion The quality of reporting of systematic reviews of surgical randomized trials has improved in the past 10 years. Some information relating to specific surgical interventions is, however, still reported poorly.


Introduction
A systematic review (SR) aims to collate all empirical evidence that fits prespecified eligibility criteria to answer a specific research question 1 . Explicit SR methods are available to identify gaps in current research, minimize bias, and provide reliable and timely conclusions 2,3 . SRs have become essential to clinical guideline development and health policy decision-making 4,5 . A SR, however, may be of limited use in clinical practice if reporting is incomplete 6,7 .
To improve the completeness in reporting of SRs and meta-analyses, the Quality Of Reporting Of Meta-analyses (QUOROM) statement 8 was developed in 1999, followed in 2009 by the PRISMA statement 9 . The PRISMA statement is a reporting guideline designed to offer a standard approach for review authors to prepare and report a SR transparently and reproducibly. Having been developed originally to aggregate outcomes from therapeutic trials, extended versions of PRISMA have been developed in relation to diagnostic test accuracy reviews 10 , harms 11 , protocol 12 , equity 13 , network meta-analyses 14 and complex interventions 15 .
Surgical intervention is more complex than simple comparisons between drugs. Special items are needed, related to surgical technology, devices, quality control measures related to the surgical procedures, and the surgeons' experience, for example.
There has been no cross-sectional study of the reporting quality of SRs of surgical RCTs with special items relating to surgical interventions. The present study was undertaken as a cross-sectional survey to investigate the extent to which SRs of surgical randomized studies adhered to the modified PRISMA checklist that combined items with intervention details. It was also investigated whether the quality of reporting had improved by comparing 2007 and 2017, with consideration of factors associated with the quality of SRs.

Methods
A study was included if it was a SR with meta-analysis of randomized trials published in 2007 or 2017, in English, that compared a surgical intervention with other interventions (another surgical intervention or a non-surgical treatment). A SR was defined as described in the Cochrane handbook (version 5.1.0) 1 , as an article with clearly stated objectives with an explicitly reproducible methodology, systematic search methods, assessment of the validity of included studies and methods of synthesis. The definition of a surgical intervention has been described elsewhere 16 .
Narrative and other types of review were excluded along with systematic reviews that included crossover trials, individual participant data (IPD) meta-analyses and network meta-analyses. Studies testing injection, acupuncture or diagnostic interventions were not eligible, and those that tested the effects of medical therapies in patients undergoing surgical procedures, such as antibiotics or adjuvant treatments in cancer, were also excluded.

Data resource and study procedure
Studies published in 2007 and 2017 were identified via PubMed. The search strategy was based on MeSH (Medical Subject Heading) terms and their variants (Appendix S1, supporting information). Two reviewers, trained in trial and SR methods, screened abstracts and full texts for eligibility, and abstracted data from eligible studies with detailed introductions. Disagreement was resolved by a third reviewer if necessary.

Data collection
The following general information was collected from each eligible study: number of authors; number of studies included in the SR; total number of participants included; number of databases searched; type of review (Cochrane or non-Cochrane); reference to PRISMA or QUOROM statement; involvement of a methodologist (such as a statistician or epidemiologist); country; journal type (general or surgical journal); type of comparison; specialty; type of funding (profit, non-profit, not funded or not reported); and significance of the primary outcome (yes, no or unclear). A primary outcome was selected according to published studies 17 : if a SR specified a primary outcome, this was chosen as the primary outcome in the present analysis. Where more than one primary outcome existed, the first one reported in methods was selected. If no primary outcome was specified, the first outcome reported in the abstract was chosen.
The items in the PRISMA checklist were used to assess reporting quality. The checklist consists of 27 items, and quality of reporting was assessed against these 27 items for each study. One point was allocated if the study met the requirement for a specific item, and no points if it did not. Each item was determined with the option of 'yes' or 'no'. Given that some items of information, such as surgical technique, preoperative care, postoperative care, rehabilitation protocol, devices and surgeon experience, are important to the application of SRs, these specific items were considered under the headings of 'data items' and 'summary of evidence' (Appendix S2, supporting information). A score ranging from 0 to 27 was thus developed.   18 . The variance inflation factor was used to explore for multicollinearity among studies 19 . SPSS ® version 22.0 (IBM, Armonk, New York, USA) was used to conduct the statistical analysis.

Results
A total of 8150 studies were identified. After title and abstract screening, 626 studies were potentially eligible. On full-text screening, 204 systematic reviews were finally included (Fig. 1). References in the included studies are shown in Appendix S3 (supporting information). The general characteristics of the included SRs are shown in Table 1. The median number of participants included was 1149 (range 55-27 807) and the median number of databases involved was 4 (1-15). Some 32 mentioned authors (15⋅7 per cent) were affiliated to a department of epidemiology, statistics or evidence-based medicine, or a methodologist was listed in the acknowledgements section. Some 43 SRs (21⋅1 per cent) were Cochrane reviews, 61 (29⋅9 per cent) were published in surgical journals, 177 (86⋅8 per cent) compared alternative surgical interventions, 56 (27⋅5 per cent) reported the funding source and 43 (21⋅1 per cent) received no funding. Studies published in 2017 included more authors than SRs published in 2007 (median 6 versus 4 respectively; P < 0⋅001), although they included fewer studies (7 versus 10; P = 0⋅013). Fewer Cochrane SRs were found in 2017 (16⋅8 versus 31 per cent; P = 0⋅021), although the number of SRs of surgical RCTs had increased in some countries, notably China. SRs from 2017 were more likely to declare clearly that they followed the reporting standards of the PRISMA or QUOROM checklist (57⋅3 per cent versus 10 per cent in 2007; P < 0⋅001). Table 2 and Fig. 2 show adherence to the PRISMA checklist. Interobserver agreement on PRISMA was good (κ statistic = 0⋅78). Of the 27 items, 15 (56 per cent) were reported adequately. Three items were reported less than 50 per cent of the time: accessibility of a review protocol and registration information (29⋅9 per cent); statement of all variables where data was sought, especially the description of intervention details (40⋅2 per cent); and description of funding source for SRs (48⋅5 per cent).

Discussion
This cross-sectional survey of 204 SRs of surgical randomized trials has demonstrated that compliance with the PRISMA checklist was moderate. Only half of the reporting items for SRs were reported adequately. Some items, such as registration, adequate description of intervention and funding, were missing regularly from many studies. The reporting of protocols and registration in SRs is much lower than in randomized trials 16,20 -25 , reflecting compulsory requirement that a clinical trial must be registered before publication by the International Committee of Medical Journal Editors 26 .
Surgical intervention details, such as devices, nursing care, anaesthetic management and surgeon experience that may be critical to the inferences drawn from SRs, especially SRs with surgery-related primary outcomes, were not well reported. Only 40⋅2 per cent of SRs included an adequate description of surgical interventions, and just 60⋅3 per cent discussed the effect of interventions on main outcomes. This places significant limitations on the value of SRs relating to these trials. An expert consensus 27 to improve the consideration and description of interventions in SRs was published in 2017. It recommended that authors of SRs should consider intervention details when planning, conducting and reporting SRs, and suggested that review authors should provide a table summarizing the intervention details for each study 28 .
A limited number of SRs reported additional analyses (such as subgroup analyses or meta-regression), but heterogeneity, led by clinical and methodological diversity, existed in all reviews, as noted elsewhere 29,30 . Review authors should consider the feasibility and appropriateness of subgroup analyses or meta-regression at the time of planning.
Despite being reported inadequately, the overall quality of reporting of SRs improved between 2007 and 2017. This improvement may be explained by the wide adoption of the PRISMA statement, increased awareness of PRISMA, and instructions from scientific journals regarding manuscripts relating to SRs 31,32 . Completeness of reporting was superior from Cochrane compared with non-Cochrane SRs, possibly owing to rigorous strategies in the editorial process and the absence of word limits 33 . The results of the present study are similar to those that included a wider range of study designs 34 -36 . There are several strengths to the present study. A validated effective search filter was used to identify all SRs published in 2007 and 2017 in PubMed, leading to greater generalizability of the findings. Rigorous methods, including explicit eligibility criteria, double-checking of study screening and data extraction, were used.
The study also has limitations. Authors' affiliation to a methodological department was determined only by the information provided on the title and acknowledgement pages. This approach may not have captured fully the expertise among authors regarding epidemiology or statistics. Items were reported as included or absent, with no qualitative assessment. The exclusion of narrative SRs, rapid reviews, overviews of reviews, and SRs with IPD meta-analyses or network meta-analyses means that the present results should not be considered generalizable to these other study designs. The present study also considered only SRs published in English.
Despite these limitations, the quality of reporting in SRs of surgical randomized trials has improved in the past 10 years. The overall findings, however, make a strong case, in particular, for the description of interventions to improve the usefulness of surgical SRs.

Supporting information
Additional supporting information can be found online in the Supporting Information section at the end of the article.