Lean Principles to Improve Quality in High-Throughput COVID-19 Testing Using SwabSeq: A Barcoded Sequencing-Based Testing Platform

Abstract

Objective

To describe and quantify the effect of quality control (QC) metrics to increase testing efficiency in a high-complexity, Clinical Laboratory Improvement Amendments–certified laboratory that uses amplicon-based, next generation sequencing for the clinical detection of SARS-CoV-2. To enable rapid scalability to several thousands of specimens per day without fully automated platforms, we developed internal QC methods to ensure high-accuracy testing.

Methods

We implemented procedures to increase efficiency by applying the Lean Six Sigma model into our sequencing-based COVID-19 detection.

Results

The application of the Lean Six Sigma model increased laboratory efficiency by reducing errors, allowing for a higher testing volume to be met with minimal staffing. Furthermore, these improvements resulted in an improved turnaround time.

Conclusion

Lean Six Sigma model execution has increased laboratory efficiency by decreasing critical testing errors and has prepared the laboratory for future scaling up to 50,000 tests per day.

next-generation sequencing, quality control, quality assurance, Lean Six Sigma, SARS-CoV-2, COVID-19 screening

Demand for robust screening for the SARS-CoV-2 virus has increased during the COVID-19 pandemic and with the lockdown measures; such screening will be needed to enable the reopening of society and provide surveillance for outbreaks and for future pandemics. The need for a highly scalable and high-throughput screening test led to the development of SwabSeq.¹ Unlike reverse-transcription quantitative polymerase chain reaction (RT-qPCR) methods, SwabSeq uses barcoded primers to amplify viral RNA and then pools the barcoded amplicons for detection of the virus using next-generation sequencing. This quantitative readout allows for improved accuracy and sensitivity, even with unpurified specimens, without sacrificing the limit-of-detection of the assay.¹ One key aspect of SwabSeq is that it is flexible and rapidly expandable to take advantage of an existing genomic core facility infrastructure that was underutilized at many universities at the start of the pandemic. Therefore, maintaining the flexibility of such core facilities can allow for the rapid deployment of massive testing protocols in the early phases of a pandemic without overwhelming clinical and health care settings.

To meet the demands of COVID-19 population screening, we implemented operational efficiency methods in a newly deployed Clinical Laboratory Improvement Amendments laboratory, the University of California Los Angeles (UCLA) SwabSeq COVID-19 Testing Laboratory, using the Lean Six Sigma model. The testing process was and continues to be optimized through the application of the model’s 5 phases: define, measure, analyze, improve, and control.² At the deployment of testing in November 2020, we made plans to scale up our capacity to >10,000 tests per day.

In this article, we outline our individualized quality control plan (IQCP)³ and our use of Lean principles to improve our workflow as we scaled up to >4,500 tests per day over a 3-month period by optimizing critical steps in our process (Lean principles represent a 5-step thought process to improve operational processes, and include identifying value, mapping the value stream, creating a flow, establishing pull and seeing perfection: https://www.lean.org/WhatsLean/Principles.cfm). In addition, we suggest fixed times for processes that are value-adding and non-value-adding. In this way, the steps necessary to scale up, such as hiring and training more staff, in addition to increasing consumable inventory, can be anticipated.⁴ Although few laboratory tests leveraging next generation sequencing are performed at significant scale, we have developed flexible, non-automation-dependent processes that can be rapidly deployed for future surveillance needs.

Methods

To further the efficiency of our testing system, we built upon the existing quality control (QC) measures by using the Lean Six Sigma model² and its tools. Three sets of variables were defined for the entirety of our process: necessary but non-value-adding operations (NNVAs), non-value-adding operations (NVAs), and value-adding operations (VAs). The NNVAs include heat inactivation, making swabseq reaction master mix, the time on the thermal cycler, and the time needed for sequencing. The NVAs include inventory, kit assembly, collections, and accessioning. The VAs include pipetting and library preparation. We used these variables to execute Lean Six Sigma’s analysis phase by calculating our turnaround time (TAT) and determining how to produce more test results simply by hiring more staff. The QC was validated using tools such as weekly huddles and QC forms.

Results

A value stream map was created to identify key people, resources, activities, and information flows required to deliver test results (Figure 1).² The SwabSeq test begins with materials that are received from the supply chain being added to the running inventory with anywhere from a 2- to 6-week lag time. Critical supplies are warehoused on site, maintaining a capacity for scale. The SwabSeq customer service team is responsible for managing kit assembly and works with staff from our testing clients to coordinate delivery of test collection materials for their sites. Each organization can use nasal swabs or neat saliva specimens that are collected into the same receptacle: a double-barcoded, internally threaded collection tube. Using PreciseQ Technologies, Inc., an electronic health care operations system, our clients can directly place order information for patient tests and we can auto-upload results. Specimens arrive racked in 96-well format on ice at the UCLA SwabSeq COVID-19 Testing Laboratory by shipment services or by client-supplied couriers. Each rack of specimens is accessioned on a 96-specimen scanner into the SwabSeq laboratory-developed database.

Figure 1.

SwabSeq value stream map showing the process of testing from supply chain to specimen collection, testing, and reporting results. Each heart represents the number of staff members required for that step. We also describe the timing for each run to indicate the feasibility of running 3 runs per day with minimal staff.

SwabSeq Testing Platform

After accessioning, specimens are heat-inactivated in a circulating water bath at 85°C to 95°C for 30 minutes. Simultaneously, another staff member will make swabseq reaction mastermix consisting of TaqPath 1-Step RT-qPCR swabseq reaction mastermix CG (Thermo Fisher), a synthetic RNA S2 standard developed and manufactured by Octant Bio, and Ambion Nuclease Free Water. After heat inactivation, the pipetting event can take place. After all specimens have been pipetted, each Applied Biosystems MicroAmp EnduraPlate 384-well plate is placed on a Thermo Fisher Veriti 384-well thermal cycler for an endpoint PCR with 50 cycles for 1 hour and 20 minutes.

SwabSeq currently uses 1536 unique dual indices (UDI), separated into four 384-well plates.¹ The primer sets were designed by Octant Inc. and manufactured by Integrated DNA Technologies. The 384-well plate is divided into 4 quadrants of 96 specimens. The 96-well racks received by the laboratory are designated as either A, B, C, or D. The letter determines which quadrant the rack is pipetted into and can be identified by specific water tube placement and our workbook file (Figure 2). Two specimen tubes containing nuclease-free water are placed into each of these 96-well racks in a location based on the assigned primer set, quadrant, and time of testing (Supplemental Figure 1).

Figure 2.

The 384-well plate organizational setup using four 96-well racks. Each 384-well plate is made up of 4 racks of specimens, each with 96 tubes. Each rack must be oriented properly at the pipetting stage and at the scanning stage to ensure that the correct specimen tube is associated with the correct plate and location.

Finally, to clearly distinguish each 384-well PCR plate, we designate a specific color to ensure that no duplicates of the same molecular indexes are sequenced simultaneously. The water tube barcodes and location are recorded during the scanning step. Therefore, during data analysis we can confirm that the expected locations of the water tube barcodes match their locations on the PCR plate. If the location of the water tubes (aka, negative control specimens) does not match the water tubes’ expected location on the plate, we can definitively see that there has been an error in either pipetting of the plate or the scanning of the plate. This simple and low-tech visualization method allows us to identify potential plate misorientation errors before reporting results.

Library preparation for the Illumina sequencers can begin as soon as the thermocycler program is completed. The Illumina NextSeq or MiniSeq can then be run, with results de-multiplexed using a continuously running informatics pipeline, developed in-house.¹ The informatics pipeline creates a series of plate visualization heatmaps for the interpretation of results and quality control. After reviewing the results, staff manually upload them into an internal network drive where an internally developed script automatically uploads results into PreciseQ every 5 minutes. Upon this upload, results are sent to the patient’s preferred method of contact and the California Department of Public Health.

To interpret results, the SwabSeq automated analysis pipeline outputs a heatmap with the read counts for each of the amplicons that are sequenced associated with their plate and well position.¹ The 3 possible amplicons we detect correspond to PCR products that represent the S2 (virus), S2-standard (internal control), and RPP30 (human input control). Therefore, for a positive virus specimen, we would expect to see a high read count of the S2, and RPP30 amplicons. The absence of reads in the RPP30 (Figure 3, dark purple-grey) indicates that no human specimen is present in that well. The absence of reads in S2-standard suggests that the well did not have sufficient S2 primer or swabseq reaction or that there was a significant PCR inhibitory effect on amplification. Finally, the presence of reads for the S2 amplicon of the SARS-CoV-2 virus suggests that there is virus present; however, a final determination of a positive or negative result is based on the ratio of the number of S2/S2-standard reads that must be above our designated threshold from internal validation.

Figure 3.

(A) Each run has a designated control water tube placement with prevalidated water tubes (id_tubes). The location is indicated by the green color (true), which confirms that these tubes were present at the expected orientation (see Supplemental Figure 1). In this example, we have two 96-well racks that were mirrored, meaning that they should have the same water tube placement (A) and location with low RPP30 reads on heat map (B). The location of the negative control water tubes within the 96-well plate do not match between plate 1B but do match between plate 4B. Plate 1 had a misorientation, resulting in a 180º plate flip identified by the rotated location of the water tubes. Comparing the S2 virus read heatmap of both plates, the plate flip is further realized because the locations of positive specimens are altered. The colors expressed in the log10 count correspond to the high presence of amplicons, purple equating to a low read count and yellow equating to a high read count. The S2 amplicon is the viral SARS-CoV-2 gene present in the specimen. The S2-standard amplicon is SwabSeq’s in vitro RNA standard to normalize viral read counts within each well. The RPP30 amplicon serves as a human-input control to determine whether a patient specimen is present in each well.

Quality Improvement Using Lean Six Sigma

Improvements to our testing process were implemented after the identification of errors that occurred during testing. Before the first scale-up in the laboratory, our minimal QC resulted in errors such as plate flips and the incorrect placement of water tubes. One such error was identified in our early testing (Figure 3). Here, we observed that for a plate that was run in duplicate, one plate was misoriented during the pipetting event but not during scanning. Therefore, we observed that the location of the id_tubes (Figure 3A) did not match the locations with no RPP30 reads (Figure 3B, red circles). Therefore, the Plate 1B rack was oriented incorrectly, causing a 180° flip of each specimen and thus a misidentification of specimens with their corresponding results. In addition, the location of high viral reads (yellow/orange) seems to be flipped between plates 1 and 4 (Figure 3B). The proper placement of water tubes provides valuable information that links the pipetted specimens to the plate to the specimen identifier and allows us to detect errors at the scanning and pipetting stage (Figure 3).

To alleviate these errors, the improve phase of the Lean Six Sigma model² was applied. We assigned specific roles to staff to ensure that quality system cross-checks were completed during each stage of the testing process. The staff member assigned the role of “support” is responsible for accessioning, heat inactivation, and setting up the pipetting event. This includes the placement of water tubes. The support role also ensures that plate flips, specimen mis-scans, and incorrect water tube placement are minimized by visually checking each orientation. The second role is that of the “pipettor.” This staff member is expected to make swabseq reaction master mix for the runs, pipette during the pipetting event, and place the 384-well plate onto the thermal cycler. After the run is set up, the pipettor verifies that each of the tasks performed by the support role is correct before progressing with the pipetting process.

To further minimize plate flips and specimen mis-scan occurrences, rack orientation checks and A1 specimen position tracking are implemented. The proper rack orientation is confirmed by the support role employee, who checks that the A1 position on the 96-well rack and the A1 position on the plate holder of the pipetting instrument match. In addition, this staffer ensures that the quadrant being plated corresponds to the correct position of the 384-well plate primer set to prevent manual plate flips. The A1 specimen position tracking form confirms that the physical tube in the A1 position and the barcode scanned into the workbook for the A1 position match, verifying that an informatic plate flip has not occurred (Supplemental Figure 2). These elements are all checked both at the time of pipetting and after all pipetting is completed, allowing 2 independent checks of the plate orientation.

Finally, because of the high volume of plates per run, we developed a visual management system through a custom-designed 3-dimensional rack template to ensure that water tube placement was in the correct row and column⁵ and thereby reduce errors in the location of water tube placement. To further verify the correct water tube placement, water tube QC forms were also added (Supplemental Figure 2).

Control Phase of Lean Six Sigma

The control phase of Lean Six Sigma² was accomplished through the creation of QC variances, clinical laboratory scientist (CLS) review of QC testing forms (Supplemental Figure S2), development of a validated training program, and weekly staff huddles. The QC variances are given after the review of QC testing forms by CLS or CLS trainees if errors are identified. The use of QC variances allows staff to identify errors and ensure that recurrence of that error is minimized. New training protocols ensure that newly hired staff are fully trained in all roles and are aware of the QC systems implemented and their importance. Weekly huddles allow for the discussion of roadblocks faced in the laboratory and how to overcome them. During these huddles, metrics are discussed to collaboratively identify sources of testing error and to implement fixes to increase the efficiency of the laboratory.

After the execution of the identification and control phases of Lean Six Sigma, we moved on to the measurement phase to determine how the implemented solutions impacted the efficiency of the testing process. The following variables were measured: average times of pipetting, total TAT from accessioning to result, the percentage of correct water tube placement, and the percentage of signed orientation checks. Average pipetting time continually decreased from 1.1 hours in December 2020 to 0.8 hours in March 2021 (Table 1). TAT also decreased from 17 hours in December 2020 to 16 hours in March 2021. In January 2021, we had an increase in specimen volume during a scale-up, altering our pipetting time and TAT. The percentage of water tube placement increased overall, but in March 2021 new staff training accounted for the small decrease. Improvements in orientation checks were also seen as the months progressed.

Table 1.

Performance Metrics After the Implementation of the Lean Six Sigma Model

Month	Pipetting Time	Total TAT from Accession to Result	Water Tube Placement, %	Orientation Checks Signed, %
December 2020	1.1	17	95 (10.45/11)	88
January 2021	1.2	18	95.4 (41/43)	100
February 2021	1.1	18	97.7 (43/44)	96
March 2021	0.8	16	95.7 (45/47)	99

Times listed are averages in hours.

The acceptable TAT from accessioning to result is 48 hours, and the goal TAT is 24 hours. Comparison of the January 2021 TAT (Figure 4A) and March 2021 TAT (Figure 4B) indicated that most runs resulted within our goal TAT of 24 hours. March 2021 had an influx of specimens, further indicating that Lean Six Sigma optimization² improved efficiency.

Figure 4.

TAT improves after implementation of improved QC metrics. Our goal TAT is 24 hours from when specimens are accessioned to when they indicate results, which is represented by the blue line. Comparing the January 2021 TAT (A) to the March 2021 TAT (B), an increase in efficiency is observed after an increase in specimens and implementation of Lean Six Sigma. QC, quality control; TAT, turnaround time.

Discussion

Our Lean Six Sigma² approach identified common points of error within our testing processes. We developed quality control metrics in our process and analytics with the placement of water tubes (Supplemental Figure S1), several layers of independent checks in our testing processes (Supplemental Figure S2), and the development of a 3-dimensional printed model to ensure the correct placement of these tubes. Our approaches have improved efficiency and TAT within our laboratory.

To continue scaling up, it was important to identify variables in accordance with Lean Six Sigma Optimization guidelines.² In our lab, we perform up to 3 runs per day during operational hours of 7:00am to 8:30pm. Some of the times in our assay are fixed, such as those for heat inactivation, the thermal cycler, and the sequencer run, adding up to 295 minutes. Accounting for an additional 50 minutes for accessioning and resulting, the fixed time is 345 minutes. The value added events during pipetting takes approximately 65 minutes per run, and library preparation takes 60 minutes per run. Therefore, a total of 470 minutes are thus allocated for total production, encompassing pipetting and library preparation. At this rate, with 1 staff member in the support role and 2 in the pipettor roles, we can perform 3 runs, each of which can test up to 1,504 specimens. Therefore, 4,512 tests can be conducted across 3 runs. By creating 2 teams, each team having 1 staff member in the support role and 2 staff members in the pipettor roles, laboratory staff can conduct a total of 9,024 tests per day.

Each team will follow the same processes, but by increasing staffing, the results will be doubled. At the writing of this article in June of 2021, the UCLA SwabSeq COVID-19 Testing Laboratory has conducted over 140,000 tests, with 6 full-time and 6 part-time testing staff, operating Monday through Friday from 7:00am to 8:30pm.

One key aspect to the SwabSeq approach is that deployment did not require full-scale automation to achieve 10,000 specimens per day. Our testing protocols require sequencing and liquid handlers as part of the critical infrastructure. However, our process is distinct enough from traditional RT-qPCR approaches that our supply chain has remained robust throughout the pandemic. Our assay uses 1 pipette tip per specimen when running extraction-free protocols, which decreases our reliance on these critical consumables that continue to be limited across clinical and research laboratories.^6,7

Conclusion

Herein, we have discussed the rapid deployment and scaling of our high-throughput COVID-19 testing program. Lean Six Sigma approaches² were key to developing a robust IQCP program to improve the efficiency of our testing processes and decrease critical laboratory errors. Although we have found solutions to the issues encountered in our first scale-up to a capacity of up to 3,000 specimens per day, we are continually adding fail-safes to this procedure.

Abbreviations

Abbreviations
 
• QC

  quality control

 
• RT-qPCR

  reverse-transcription quantitative polymerase chain reaction

 
• UCLA

  University of California Los Angeles

 
• IQCP

  individualized quality control plan

 
• NNVA

  necessary but non-value-adding operation

 
• NVA

  non-value-adding operation

 
• VA

  value-adding operation

 
• TAT

  turnaround time; CLS, clinical laboratory scientist.

Acknowledgments

V.A.A. supported by NIH (DP5OD024579).

Author notes

These authors contributed equally.