To the Editor—We welcome the novel approach adopted by Mistry et al. in attempting to identify urgently needed biomarkers for relapse after treatment of tuberculosis . However, although this work is methodical and clearly a promising first step, we believe that aspects of the study design weaken some of the claims made for this new approach.
The authors state that there is currently no reliable method of predicting relapse. However, using a similar linear discriminant analysis of radiological and bacteriological data from 846 patients enrolled in 2 prospective clinical trials, Aber and Nunn showed that 3 factors (radiological cavitation and 2-and 3-month culture results) contributed significantly to prediction of subsequent relapse . Although imperfect, 2-month culture status is now widely used as such a predictor of relapse, and its performance has been evaluated in numerous other trials .
By contrast, the authors understandably chose to adopt a matched cross-sectional design in their pilot study because recurrence is such a rare event after modern therapy. However, the method of selection of subjects and the timing of sampling add to the difficulties of interpretation of this retrospective approach. In particular, patients with recurrent disease, presumably attributable in most cases to reinfection, were sampled 18 months after treatment for their latest episode was completed. Because the putative surrogate end point was measured after the reference end point had occurred, we must assume that samples taken long after treatment has finished are in fact representative of those that would in practice need to be taken before or at the end of treatment to be of any practical use. The temporality that would need to hold in clinical trials has been reversed, and we must therefore remain uncertain about how the biomarker would perform in prospective studies.
More seriously, in randomized trials, reinfection events should be randomly distributed between the treatment arms and carry no information about the efficacy of treatment. The strong determination of “recurrence” by host factors that the authors seem to be proposing would only appear to support this traditional view. By contrast, true “ relapses” are believed to be causally related to the efficacy of therapy, yet it would appear that few such patients were included in the study of Mistry et al. Thus, expression patterns of host genes persisting after clinical cure could reasonably be expected to predict reinfection events in a matched design with a high proportion of the data set devoted to training. This result, however, gives little assurance that the same discriminant function could be successfully applied directly to new data sets with greater inter-individual variability in expression and a lower proportion of reinfections. It is even possible that different functions or sets of genes would be selected for different study populations, raising the question of how these composite biomarkers should be compared.
In general, for a biomarker to be used as a surrogate end point for the evaluation of new treatments in a clinical trial, differences in the biomarker should reliably predict the difference in clinical outcome between 2 treatment groups (trial level surrogacy) . Showing even perfect association between a biomarker and the clinical end point (individual level surrogacy) is neither sufficient nor necessary to qualify the biomarker as a useful trial level surrogate . To draw an analogy with a related field, it is well known that CD4 cell count is a useful individual-level surrogate for disease progression in HIV-infected individuals, but several studies have shown that change in CD4 cell count during antiretroviral therapy is a poor trial-level surrogate [6–8].
The authors describe an approach that may be capable of discriminating between patients who are cured and those who will later relapse, and this could ultimately be useful in clinical practice. To “facilitate clinical trials of new chemotherapeutic agents or shortened treatment” [1, p. 369] however, it must also be demonstrated that the biomarker fully captures the effect of treatment on the clinical end point. It is encouraging that new biomarkers are in development that could ultimately improve on intermediate bacteriological results in tuberculosis trials, but we believe that more consideration needs to be given to the methods by which these new tools are to be evaluated.