The Fragility Index: a P-value in sheep’s clothing?

Extract

This editorial refers to ‘How robust are clinical trials in heart failure?’^†, by K.F. Docherty et al., on page 338.

Randomized controlled trials (RCTs) are designed to assess objectively the safety and efficacy of a specific intervention. While public reporting of the study design, conflict of interest declarations, and the peer review process strive to fortify study findings, the veracity of published research studies has been called into question.¹ Recently, the Fragility Index (FI) has been introduced as an intuitive measure of the robustness of RCTs.² This new statistic has been recently reported in the critical care setting,³ and studied and advocated for heart failure in the paper by Docherty et al.⁴ In this Editorial, we will provide a brief explanation of the FI and express why caution needs to be exercised when interpreting FI.

The FI is a statistical summary of an RCT that utilizes 1:1 randomization and an outcome measure that can be categorized into two levels (e.g. 30-day mortality, yes or no). When a study reaches a statistically significant difference (and the sample size per group is fixed), one can show that the number of positive responses differs between the groups. Without loss of generality, assume that a study is comprised of a novel intervention and a control condition and that fewer events will be observed in the intervention group (i.e. the intervention provides a protective effect to the patients). Suppose that the study results suggest that the intervention is protective and that the result is statistically significant. Upon careful observation, it is noted that the number of events is reasonably small and that in absolute number there is not a large difference between the two study groups. One might naturally ask the question ‘how would the results be interpreted if one of the non-events in the intervention group switched to being an event?’ Would the RCT result reach the same statistical conclusion? What if two patients switched? What if three? Etc. Table 1 shows this general process in the context of a clinical trial with n = 100/group. The original study would have response rates of 6% (6/100) and 17% (17/100) for the intervention and control, respectively. The FI is defined as the minimum number of reversals that need to occur for the result to be no longer statistically significant at the alpha = 0.05 level of significance using Fisher’s exact test.² In the context of the table, the FI would be estimated to be 2.