Generic Protocols for the Analytical Validation of Next-Generation Sequencing-Based ctDNA Assays: A Joint Consensus Recommendation of the BloodPAC’s Analytical Variables Working Group

Abstract Liquid biopsy, particularly the analysis of circulating tumor DNA (ctDNA), has demonstrated considerable promise for numerous clinical intended uses. Successful validation and commercialization of novel ctDNA tests have the potential to improve the outcomes of patients with cancer. The goal of the Blood Profiling Atlas Consortium (BloodPAC) is to accelerate the development and validation of liquid biopsy assays that will be introduced into the clinic. To accomplish this goal, the BloodPAC conducts research in the following areas: Data Collection and Analysis within the BloodPAC Data Commons; Preanalytical Variables; Analytical Variables; Patient Context Variables; and Reimbursement. In this document, the BloodPAC’s Analytical Variables Working Group (AV WG) attempts to define a set of generic analytical validation protocols tailored for ctDNA-based Next-Generation Sequencing (NGS) assays. Analytical validation of ctDNA assays poses several unique challenges that primarily arise from the fact that very few tumor-derived DNA molecules may be present in circulation relative to the amount of nontumor-derived cell-free DNA (cfDNA). These challenges include the exquisite level of sensitivity and specificity needed to detect ctDNA, the potential for false negatives in detecting these rare molecules, and the increased reliance on contrived samples to attain sufficient ctDNA for analytical validation. By addressing these unique challenges, the BloodPAC hopes to expedite sponsors’ presubmission discussions with the Food and Drug Administration (FDA) with the protocols presented herein. By sharing best practices with the broader community, this work may also save the time and capacity of FDA reviewers through increased efficiency.


Introduction
Circulating tumor DNA (ctDNA) is genomic material shed by apoptotic and necrotic tumors into peripheral circulation (1). It typically represents only a small portion of cell-free DNA (cfDNA) present in the blood, which can originate from many different sources such as infectious organisms, fetal DNA during pregnancy, and genomic DNA from white blood cells (2). The presence of tumor-associated mutations allows highly specific discrimination of ctDNA from normal DNA. However, the relatively low abundance of ctDNA compared to cfDNA means that there are few mutant molecules present, necessitating well-characterized and analytically robust ctDNA assays to ensure the reliability of molecular information.
The use of ctDNA to obtain tumor genomic information via a minimally invasive blood draw has attracted a significant amount of attention in the era of precision medicine. In addition to the ease of accessibility and reduced risk to the patient compared to tissue biopsy, several other characteristics of ctDNA make it an attractive tool for use in the clinic. First, ctDNA analysis can be performed very quickly compared to tissue analysis that has a greater number of preanalytic steps that result in increased turnaround time and can significantly delay the administration of appropriate therapy (3,4). As invasive tissue biopsy carries with it risks inherent for any surgical procedure (5,6), molecular analysis of tissue is usually limited to a single sample collected at a single time-point, which may not be sufficient to capture the dynamic nature of a tumor that is constantly evolving under selective pressure from different clinical interventions. Moreover, the heterogeneous nature of the complete population of tumor cells present in a patient means that molecular analysis across different sections of a tissue specimen may be subject to biological variation. Biopsy of a single anatomic location may also fail to capture a global picture of disease, thus missing the genomic profiles of cells present at other sites such as distant metastases. In contrast, ctDNA is easily obtainable through collection of multiple specimens corresponding to clinically important time points, such as at baseline diagnosis, after surgical resection of a tumor, at progression, or other clinically relevant time points and across the course of different therapeutic regimens. It is also hypothesized that ctDNA may better capture disease heterogeneity since tumor-derived mutant molecules that originate from all disease sites across the body can be sampled via peripheral blood draw (7).
Despite these advantages, evidence generation for use of ctDNA in clinical practice has progressed at a measured pace and there are currently only a few applications where ctDNA results are routinely used to inform a clinical decision. Specifically, it has been demonstrated that the use of liquid biopsy in treatment selection for previously untreated patients with nonsmall cell lung carcinoma leads to discovery of guidelinerecommended biomarkers at a rate that is similar to that of standard-of-care tissue genotyping, with high tissue concordance and more complete genotyping (8). Moreover, patients identified for targeted treatment using liquid biopsy have demonstrated similar therapeutic response rates to patients identified with tissue testing (5).
Clinical practice guidelines now recognize the utility of ctDNA analysis in advance of intrathoracic biopsy and tissue analysis for patients with nonsmall cell lung cancer who are potentially eligible for an EGFR tyrosine kinase inhibitor (9)(10)(11)(12). Additionally, a real-time PCRbased ctDNA test was recently approved contemporaneously with the first approved therapy targeting PIK3CA for patients with estrogen receptor-positive breast cancer who progress on adjuvant hormone therapy and are in need of a second-line option (13). However, in both cases, guidelines recommend that if a negative ctDNA result is obtained, patients should be reflexed to tissue testing. This is particularly relevant in the context of hotspot testing using quantitative PCR, which has limited sensitivity compared to technologies such as digital PCR and NGS, since a negative result does not differentiate absence of the mutation of interest from lack of sufficient ctDNA in the sample. In contrast, with NGS-based tests, tumor profiling information obtained panel-wide can often differentiate these 2 scenarios by identifying tumor-derived alterations at higher allele frequencies and suggesting true absence of mutation in a background of detectable ctDNA levels. Therefore, it follows that to minimize the number of patients who undergo invasive tissue biopsy (or potentially receive no conclusive molecular testing results at all); ctDNA assays must be well designed and capable of demonstrating robust performance for their clinical intended uses.
In addition to the selection of targeted therapies, ctDNA presents an attractive option that may have potential use as a minimally invasive biomarker to monitor disease status. For example, Kruger et al. (14) recently demonstrated that serial monitoring of ctDNA as identified by the presence of a KRAS gene mutation for patients with advanced pancreatic cancer receiving chemotherapy might be useful for early response prediction and therapeutic monitoring. The persistence of ctDNA postsurgical resection of a primary tumor may also be indicative of microscopic occult disease, which may drive disease recurrence (9,15). Liquid biopsy also has been successfully used to predict therapeutic response based on early sampling of ctDNA levels 4-6 weeks posttherapeutic intervention in lung and bladder cancer (9). However, these and other potential clinical uses of ctDNA testing are still under study and will likely require diagnostic tools, which have demonstrated unique performance characteristics. For instance, a ctDNA test used for therapeutic response monitoring will need to demonstrate reliable quantitative detection of the absolute number of mutant ctDNA molecules per unit of volume of plasma to provide the highest resolution information on dynamic changes in disease status as compared to a qualitative "mutation detected" versus "no mutation detected" readout sufficient for a targeted therapy selection test.
As ctDNA tests, particularly those based on NGS, continue to advance toward adoption into routine clinical practice, there will be a need for a standardized approach to assay validation to ensure the legitimacy of information on which clinical decisions increasingly will be based. Historically, the role of standards, guidelines, and best practices are prominent in US healthcare and in technology industries globally, and can function as vital and stabilizing entities, especially during times of disruptive change. For instance, the National Comprehensive Cancer Network V R (NCCN V R ) guidelines have helped to establish national standards-of-care, clinical best practices, along with rank-ordered category recommendations for a broad range of cancers. In the field of medical testing, the Clinical and Laboratory Standards Institute (CLSI) has developed a range of global standards to harmonize the clinical laboratory practices and the development of diagnostic devices (16). While these guidelines are informative to the practice of ctDNA testing and validation, they do not entirely address the level of stringency or the workflow challenges presented in the development of highly sensitive, ctDNA-based NGS tests, as recognized by BloodPAC in the drafting of these generic protocols.
With the release of the FDA's finalized guidance around the Breakthrough Devices Program, the BloodPAC is encouraged to see several members receiving breakthrough designation of their NGS-based ctDNA tests. However, the BloodPAC recognizes that despite this move toward innovation in the field, few ctDNA-based tests have received FDA approval to date (17). In addition, there is currently uncertainty in the oncology community about the reliability of ctDNA testing, which has been fueled by studies showing discordance with tissue genotyping (18) as well as studies showing discordance among plasma-based tests, especially for detection of low-frequency mutations (19,20). These studies highlight the necessary caution as these technologies advance toward FDA approval and more widespread clinical practice. Guidelines and best practices are crucial for the successful development of robust and accurate liquid biopsy clinical assays.
The BloodPAC Consortium also recognizes that, although analytical validation studies for tissue-based tests are, in many ways, similar to those that should be performed for ctDNA assays, several unique biological considerations must be taken into account. These considerations include the low concentration of analyte (down to several mutant molecules per 10-mL blood collection tube (BCT)), which both challenges the detection capabilities of assays as well as necessitates an increased use of contrived specimens for validation, for which functional equivalence with native ctDNA must be demonstrated. Another consideration is that normal cfDNA will also be present in abundance relative to the concentration of ctDNA and may fluctuate for a variety of biological and clinical reasons unrelated to disease status. In addition, the presence of nontumor-derived mutations in the blood as an endogenous analyte, i.e., germline or clonal hematopoiesis of indeterminate potential (CHIP) pose another challenge. Novel technology solutions have been developed for ctDNA analysis, such as the use of unique molecular identifiers to increase the sensitivity and specificity of NGS ctDNA assays (18). Preanalytical procedures must also be optimized to ensure not only the stability of the ctDNA analyte, but also to prevent the release of large quantities of wild type (WT) genomic DNA from white blood cells that could dilute the detectable signal. Finally, analysis methodologies must be optimized to enable accurate detection of signal in the background of PCR and sequencing errors.
Under this context, the Analytical Variables Working Group (AV WG), within the larger BloodPAC Consortium, approached the FDA to propose collaboration on a set of generic analytical validation protocols designed to ensure the robustness and standardization of assay validation while specifically addressing the unique challenges of ctDNA testing. Utilizing the collective participation among all BloodPAC participants including members from biotech and pharmaceutical industries, academia, and US Government scientific and regulatory sectors, the AV WG developed a series of generic analytical validation protocols ((21), see the online Data Supplement for Material 1). These generic protocols are intended to help guide the analytical validation of NGSbased ctDNA tests that yield information that can help inform treatment decision-making. Our goal is to create a standardized generic starting point for all analytical validation protocol discussions such that methods described can be adapted and applied for any NGS ctDNA assay, irrespective of test design, enrichment technology, or any other workflow component. Through this guidance, the BloodPAC seeks to accelerate and streamline the submission process for both test sponsors as well as the FDA and, most importantly, to safeguard the quality of molecular information that will be used to enable therapeutic decision-making for patients. In addition, studies described are primarily specific for treatment decision-making in patients who have already been diagnosed with late-stage solid tumors. Due to the rapid advancement of ctDNA technologies and clinical applications and the corresponding evolution of the regulatory landscape, the AV WG views this exercise as a balance between conveying current information and anticipating future practices. To date, we have created 11 generic analytical validation protocols along with 4 standard methods, which are described next.

Materials and Methods
During Q4 of 2017, the BloodPAC AV WG was formed, with leadership by James Godsey and Angela Silvestro to ensure balanced inputs from diagnostic and therapeutic sectors. The kickoff meeting for this new working group took place at the BloodPAC quarterly meeting held in December 2017. The initial step to forming this working group was to gather a group of key BloodPAC members who represented in vitro diagnostic manufacturers, and pharmaceutical, biotechnology, academic, and government organizations. This working group included 20 members and 2 cochairs and represented 11 companies and organizations.
Once the membership was established, the AV WG aligned on a common objective to provide a complete set of generic analytical validation protocols designed to provide test developers/manufacturers with a core baseline of standardized analytical validation protocols with which to document a ctDNA assay's analytical performance.
Given this objective, the AV WG also aligned on a set of key assumptions to focus the scope of the effort: 1. The protocols will be used for assays with a generic intended use. Namely, these protocols are intended for the validation of NGS-based ctDNA assays that yield information to aid treatment decision-making in patients who have already been diagnosed with advanced solid tumors. These protocols are not intended for the validation of screening assays or assays for early detection. 2. The protocols are designed for the validation of assays with a locked assay design that has been developed under design control. 3. These protocols aim to produce validation data intended to be submitted for regulatory review. 4. These protocols are designed to be technology and design independent (i.e., are not specific to one chemistry, instrument, or bioinformatics pipeline).
Following alignment on objectives and assumptions, the members met to define study designs based on best practices in the field, existing assay validation guidance documents (such as from CLSI, New York State Department of Health, and the Association for Molecular Pathology), and the unique performance challenges for NGS-based ctDNA assays. The AV WG decided to follow a standard format, proposed by the FDA's Center for Devices and Radiological Health (CDRH), in which each protocol contains an introduction, experimental design, a statistical analysis section, and an example data presentation format. In addition, a set of standard methods around sample collection and preparation were also included in the document. Biostatisticians were consulted to define the minimal test requirements, sample size, and to give guidance on the appropriate statistical analysis.
Throughout the drafting process, the AV WG regularly consulted with the FDA's CDRH to align on the approach and direction of these generic protocols. Revision 1 of the protocol document was submitted to CDRH in the form of a presubmission in August 2018. Based on the FDA's feedback, the AV WG made additional revisions and submitted Revisions 2 and 3 of the document as supplemental presubmissions to CDRH in February and July of 2019, respectively. After iterative discussions, the FDA's inputs were incorporated into the following set of protocols (see Supplemental Material 1).

PROTOCOLS AND STANDARD METHODS
A summary outline of the experimental designs for the analytical validation studies is found in Table 1 and includes general guidance on minimal sample size. A set of standard methods is also summarized in Table 2. Detailed protocols and methods, including statistical analyses methods are found in Supplemental Material 1. It is expected that assay developers will determine an appropriate sample size for demonstration of performance to support claims for each unique assay. For some studies (e.g., accuracy), the minimal sample size must be determined by the lower bound of the acceptance criteria. For other studies that establish analytical performance based on point estimates or the performance goal of the assay [e.g., limit of blank (LoB) or limit of detection (LoD)], the minimal sample size should sufficiently address the uncertainty of the performance estimates to support the claims. Unless specified, the number of lots, sites, and/or equipment, and operators per study are to be determined by the assay developer. In addition to these performance factors, new assay developers should also note that BCTs are an important component of any ctDNA assay and therefore should be selected and evaluated appropriately for each assay's intended use. The selection of an appropriate BCT should be made prior to the start of analytical validation.
The AV WG considers the work that follows as a baseline that ctDNA assay developers/manufacturers can utilize in their initial presubmission discussions with the FDA. As stated in the Methods section, these protocols are designed for the validation of assays with a locked assay design that has been developed under design control.
CHIP, which refers to the clonal expansion of white blood cells derived from a hematopoietic stem or progenitor cell that have acquired one or more somatic mutations (23), poses a unique challenge for NGSbased ctDNA assays since it can cause the apparent detection of tumor-associated somatic mutations in the plasma of healthy, aging individuals. In fact, these apparent somatic mutations are known to be the result of mutations affecting white blood cells, which may be considerably expanded in individuals of advanced age. As CHIP results in the true presence of mutant DNA molecules in circulation, it does not cause analytical false positives; however, because these mutations are not tumor-derived, detection of CHIP by a ctDNA test results in a clinical false positive liquid biopsy result (24). Therefore, clonal hematopoiesis is a confounding factor for liquid biopsy assays that necessitates differentiation between somatic and CHIP variants. In addition, these CHIP variants detected in plasma may contribute to the discordance observed between tissue and plasmabased assays (24).
One recommended approach for exclusion of nontumor-associated CHIP mutations is to sequence matched white blood cells at the same depth as plasma. Developers may also choose to address CHIP bioinformatically; as each laboratory has different methods for

Continued
Generic Protocols for ctDNA Assay Validation

Special Report
Clinical Chemistry 66:9 (2020) 1161  delineation of CHIP mutations, CHIP-filtering approaches are generally proprietary and specific to each developer's platform. One example approach is the use of computational methods to distinguish between germline and somatic variants by modeling the expected allelic frequencies of germline, somatic, and subclonal mutations using the mutant allelic fraction and sample purity (25). Developers should consider the benefits and disadvantages of each method to address CHIP carefully and weigh each in light of the assay's intended use and performance goal. For example, filtering CHIP by sequencing matched normal in parallel with ctDNA may be confounded if circulating tumor cells, which harbor tumor variants, are coisolated with white blood cells (26). Another potential difficulty of this approach is that assays optimized for ctDNA analysis may not demonstrate sufficient performance for the identification of CHIP via matched normal sequencing. Alternatively, developers who use algorithmic approaches to address CHIP must demonstrate that the specificity and sensitivity goals of the test's intended use are met.
Since CHIP may affect the false positive rate of the assay, developers should test samples used in the LoB and Reference Interval studies using the final device's CHIP-filtering method. All subsequent analytical validation studies should also be completed using the final device's method for filtering CHIP.

A NOTE ON LABORATORY DEVELOPED TESTS
The following protocols were drafted with input from the FDA and may be useful for the generation of validation data for the regulatory approval of new in vitro diagnostic assays. Developers of laboratory developed tests (LDTs) that utilize plasma-based NGS assays may also find these best practices informative. All developers should consult with a statistician to determine the sample number and study size appropriate for each assay's intended use and performance goal. Developers of LDTs may reference the following guidance, as well as those from New York State, the College of American Pathologists, and CLIA (27).

REFERENCE MATERIALS
Reference material samples should be analyzed regularly at defined run intervals to detect possible sources of error and to monitor assay performance (28). These materials should be chosen to reflect the size distribution and yield (for full-process materials that require extraction) of ctDNA derived from patient samples and should be functionally comparable to patient samples. Reference materials should be prepared carrying variants at levels that can confirm the performance of the assay for its intended use as demonstrated by the assay's analytical validation studies. Reference materials may include commercially available materials (see Supplemental Materials 2) or may be prepared as described in Table 2. Blank or "No-Template Control" samples such as water or Tris-EDTA buffer should also be included to identify possible contamination affecting the assay's laboratory workflow.

Discussion
In an effort to accelerate the approval and commercialization of ctDNA assays in the liquid biopsy field, the BloodPAC Consortium has developed, via the formal FDA presubmission process, a document defining a set of generic analytical validation protocols and standard methods customized explicitly for NGS-based ctDNA assays. Utilizing its diverse membership, which includes the FDA, the BloodPAC's AV WG's generic analytical validation protocols allow ctDNA test developers to initiate presubmission discussions with regard to analytical validation test protocols specific to their test's intended use, at a much greater level of protocol maturity and completeness. As a result, this minimizes the time spent in presubmission review (each cycle is $90 days), with the end result being the acceleration of the approval of new ctDNA assays. The BloodPAC's generic analytical validation protocols also minimize the time the FDA reviewers need to invest in the multiple rounds of reviews of presubmissions focused on analytical validation for the same reasons. This collaboration results in a win-win situation for both sides; test developers come to the FDA with protocols that are already compliant to the FDA's guidance and, in turn, the test developer gains months and/or quarters in bringing a new ctDNA to market.
The 11 protocols presented herein fulfill the need for standards and best practices by providing guidance on the execution and analysis of validation studies for NGS-based ctDNA assays, including recommended sample types, suggested sample size, and frameworks for statistical methods and data presentation. In addition, the AV WG has also provided 4 standard methods for the collection and preparation of specimens and for the production of contrived cfDNA samples in support of these protocols. These sets of protocols should be viewed as a starting point of a "living document." The protocols may need to evolve as our scientific understanding of cancer continues to improve and technology advances.
Though the primary focus of this work has been validation of ctDNA assays, which may be used to guide administration of targeted therapies and for response monitoring, the BloodPAC's AV WG will also consider assay performance characterization necessary to support novel uses for ctDNA analysis in the future. In particular, as evidence continues to increase in support of the validity of molecular measurable residual disease assessment to inform clinical decisions, the need for ultra-sensitive ctDNA detection is now paramount. Additionally, the BloodPAC Consortium is uniquely positioned to further advance the development and implementation of liquid biopsies by combining development of best practices with our expertise in data collection and management via the the BloodPAC Data Commons, an open repository for liquid biopsy scientific and clinical data (21,29). This will allow us to address additional challenges to the implementation of liquid biopsy in a clinical setting such as discordance between liquid biopsy and tissue biopsy, challenges to ctDNA detection in low tumor burden disease, and difficulties associated with use of liquid biopsy for disease monitoring. An additional area of focus also includes the evaluation of reference materials for use in cfDNA assay development and validation. The BloodPAC will continue to work with organizations in the field such as the American Association for Clinical Chemistry, Association for Molecular Pathology, and CLSI to advance these areas and develop community supported guidelines and frameworks.

Supplemental Material
Supplemental material is available at Clinical Chemistry online.
Disclaimer: This article reflects the views of the authors and should not be construed to represent policies of the FDA or NCI.
Author Contributions: All authors confirmed they have contributed to the intellectual content of this paper and have met the following 4 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; (c) final approval of the published article; and (d) agreement to be accountable for all aspects of the article thus ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved.