TO: CC: Members of the VQA Advisory Board (VQAAB) Bill Meyer Bob Coombs/Ming Chang Nicole Tobin Belinda Yen-Lieberman Joan Dragavon Urvi Parikh Jessica Fogel James Bremer Cheryl Jennings Carolyn Yanavich/Diane Costello Anita Gillis/Jenny Nguyen Heather Sprenger Ron Bosch Joe Fitzgibbon, Keith Crawford FROM: Don Brambilla, Dan Zaccaro, Amy Couzens and Barbara Do DATE: SUBJECT: Samples 26-30 of RNA035 The following is the first report on the analysis of VQA HIV RNA proficiency testing data through samples 26-30 of RNA035. Eighty-eight datasets from 69 laboratories were included in the analysis. Datasets that were generated by different kits in the same laboratory were analyzed separately. Eighty-three of the 88 datasets consisted of results from the most recent four 5-member panels (11-30 of RNA035). Dataset 11 consisted of the results of a 20-member qualifying panel that was recently tested plus 1-3 5-member panels that were tested after the 20-member panel. As usual, the composition of the panels depended upon the kit that was used in a given laboratory (Tables 1-4). Proficiency panels are scored on the total assay standard deviation of log10 recovery, accuracy, counts of negative results near the limit of detection, false positives and invalid results. The total assay standard deviation (SD) is related to the intra-assay and inter-assay standard deviations as ST=(SA 2 +SE 2 ) 1/2 where ST is the total assay SD, SA is the intra-assay SD and SE is the inter-assay SD. The intra-assay, inter-assay and total assay SD are calculated for each data set (Table 5) and are used to identify which data sets exhibit problems with precision and which component of variation is contributing to the elevated precision statistic. Sensitivity is assessed by determining the number of negative results obtained from replicates at the limit of detection for the assay in question. The limit of detection is defined as the nominal concentration at which 95% of assay results are expected to be positive (Roche COBAS AmpliPrep / COBAS TaqMan HIV-1, version 2 assay (Roche TaqMan), Abbott RealTime HIV-1 assay (Abbott RealTime), and internally developed taqman (IH) assays: 50 copies/ml; and biomerieux NucliSens EasyQ HIV RNA (EasyQ): 200 copies/ml). Specificity is assessed by monitoring the results obtained on samples that do not contain HIV. It is also expected that all results should meet established criteria for assay validity. Accuracy is assessed by comparing average(mean) log10 recovery in a laboratory with expected log10 recovery. The average is calculated using the subset of samples that is used to assess the total assay standard deviation. Median log10 recovery across all laboratories in which the same kit was used on the panels in question is treated as the expected value. Estimated kit-specific log10 values are calculated for each analysis (Table 6). Per the VQAAB s recommendation, an analysis investigating the impact of standard errors on the accuracy statistics on previous rounds of testing was conducted in April 2009. As a result of this investigation, a modified approach for the scoring procedure is implemented when the SE on a round of testing is low enough to be of concern. Under this modified approach, the observed standard error on a round of testing is replaced by the historical median of 0.080517 when the observed SE is below the median; otherwise, the observed value would continue to be used (Table 7). 1
A minimum of 7 data sets must be available in order to generate a reliable median. If fewer than 7 data sets are available for any given assay, then accuracy is assessed using modified criteria. In this case, median log10 recovery is compared to 1) the nominal concentration; 2) an overall real-time cross-platform derived median; and 3) an overall cross-platform derived median (Table 8). The overall real-time cross-platform-derived median was determined using data only from all the commercially available real-time PCR kits (specifically, Abbott RealTime and Roche TaqMan). The overall crossplatform-derived median is determined using data from all the commercially available PCR kits. All three medians are used for determining a final accuracy score. If the same penalty is assessed for 2 of the 3 medians, then that penalty is assessed (this could be a no-penalty scenario in which 2 out of the 3 p-values are > 0.05). If the results indicated three different penalties (i.e. >0.05, between 0.05 and 0.01, or <0.01) then a penalty of between 0.05 and 0.01 is assessed, which is equivalent to a PC. In this analysis, there were fewer than 7 data sets available for the IT and EasyQ assays. The modified criteria for assessing accuracy were used to evaluate accuracy in these data sets. The VQA will now be providing official scores for non-technical criteria including the timeliness of query responsiveness and the submission of data containing patient identifiable information or protected health information (PII/PHI). The inclusion of these non-technical criteria were unofficial for two rounds of testing starting with the scoring of RNA032-1 (July 2016). The official scoring of these non-technical criteria will begin with the scoring of samples including RNA032-3 (November 2016). The non-technical scoring nomenclature will help to differentiate between problems with late data submissions (T), unresponsiveness to data queries (Q) and submission of data containing PII/PHI (P). A number of 1 or 2 will be used to indicate if one (1) or more (2) of a given problem was noted. Multiple letters will indicate if multiple problems were noted in the last four analyses (C1 T,P or C2 Q ). Technical scores will be indicated as done previously (C, PC, P) and the nontechnical problems will be indicated as superscripts to the technical score. The numeric score for each round that is used for determining the cumulative rating will be based on the combined technical and non-technical criteria. Beginning in September 2017, the VQA provided both an official and an unofficial score based on reported Abbott false negative results derived from the 50cp/mL controls. The official score did not incorporate sensitivity results based on 50cp/mL samples; unofficial scores reflected false negative results as per our historical criteria. Sensitivity results were excluded from official scores due to an investigation into the increased rate of false negative results (from Abbott kits) for VQA HIV RNA proficiency testing samples with a nominal value of 50 cp/ml. As of the primary analysis for RNA panel 035, samples 6-10, the VQA has resumed official scoring for sensitivity for Abbott data starting with samples 6-10 of RNA035. Any false negatives reported for panels RNA032 or RNA033 will not be included in sensitivity scoring, and have been removed from Table 9; only data from RNA035 testing or later will be used for sensitivity scoring. Beginning in March 2018, the VQA will be providing both an official and an unofficial score based on accuracy results for the Roche cobas HIV-1 quantitative nucleic acid test. The official score will not incorporate accuracy statistics; unofficial scores will reflect accuracy statistics per the historical criteria. Accuracy results will be excluded from official scores until sufficient numbers of data sets (>7) are available to permit the estimation of within-assay median values for use in accuracy scoring. It is anticipated that this report will be discussed during our next conference call. We can be reached for questions or comments via email (dzaccaro@rti.org), telephone (919-541-6310) or FAX (919-541-5966). 2
RESULTS: ROCHE COBAS ASSAY No technical problems were identified in the 8 datasets generated with the Roche cobas HIV-1 assay (Table 9). Scores of C are recommended for the datasets generated with this assay. Trends in log10 recovery with Roche Cobas kit data are illustrated in Table 12 and Figure 3. RESULTS: ROCHE TAQMAN ASSAY Technical problems were identified in 3 of the 32 datasets generated with the Roche TaqMan assay (Table 9). Accuracy issues were noted in datasets 15 and 17. A score of PC is recommended for dataset 15 (an accuracy statistic of -3.88 was observed), and a score of P is recommended for dataset 17 (an accuracy statistic of -4.27 was observed). A failed VQA 200 copy control in the last round (35-5) was observed in dataset 24. The lab did not repeat the panel, so the 5 invalid samples have carried forward. A score of P is recommended for dataset 24. Scores of C are recommended for the other 29 datasets generated with this assay. We continue to monitor log10 recovery for laboratories using the Roche TaqMan Assay (Table 10; Figure 1) and will continue to share data with Roche Molecular Systems (RMS) on the performance of their assay; new analyses looking specifically at log10 recovery between data sets generated with AmpliPrep vs. High Pure manual extractions have been added to this monitoring. RESULTS: EASYQ ASSAY No technical problems were identified in the 2 datasets generated with the EASYQ assay (Table 9). Scores of C are recommended for the datasets generated with this assay. RESULTS: GENEXPERT ASSAY No technical problems were identified in the 1 dataset generated with the GeneXpert assay (Table 9). A score of C is recommended for the dataset generated with this assay. Trends in log10 recovery with GeneXpert kit data are illustrated in Table 13 and Figure 3. RESULTS: ABBOTT REALTIME ASSAY Technical problems were identified in 2 of the 45 datasets generated with the Abbott RealTime assay (Table 9). One false positive result was noted in dataset 54 which was a carryover from the a previous round (35-4), but the laboratory did not submit a repeat panel. A score of PC is recommended for this dataset. A failed VQA 200 copy control (VQA200 reported as 60 copies/ml) was observed in dataset 75. This invalidates all 5 samples. A score of P is recommended for dataset 75. Scores of C are recommended for the other 43 datasets generated with this assay. Trends in log10 recovery with Abbott kit data are illustrated in Table 11 and Figure 2. The Abbott mean log10 recovery continues to be relatively flat. No problems with query unresponsiveness or the submission of data that included patient identifiable information (PII) or protected health information (PHI) were noted in any of the data sets included in this analysis. It is anticipated that this report will be discussed during our next conference call. We can be reached for questions or comments via email (dzaccaro@rti.org), telephone (919-541-6310) or FAX (919-541-5966). 3
TABLE 1. COMPOSITION OF SAMPLES 11-15 OF RNA PROFICIENCY PANEL RNA035*. NOMINAL CONCENTRATION (COPIES/ML) RT 0 1 50 1 850 1 4,250 34,000 1 238,000 1 TABLE 3. COMPOSITION OF SAMPLES 16-20 OF RNA PROFICIENCY PANEL RNA035*. NOMINAL CONCENTRATION (COPIES/ML) RT 0 1 50 1 850 1 4,250 1 34,000 1 238,000 TABLE 4. COMPOSITION OF SAMPLES 21-25 OF RNA PROFICIENCY PANEL RNA035*. NOMINAL CONCENTRATION (COPIES/ML) RT 0 1 50 1 850 1 4,250 34,000 1 238,000 1 TABLE 4. COMPOSITION OF SAMPLES 26-30 OF RNA PROFICIENCY PANEL RNA035*. NOMINAL CONCENTRATION (COPIES/ML) RT 0 1 50 1 850 1 4,250 1 34,000 238,000 1 *Prior to panel RNA035, the OT configuration was used by labs using the EasyQ assay; the RT configuration was used by labs testing the Roche Cobas, TaqMan, In-House, GeneXpert or Abbott RealTime assays. Starting in panel RNA035, all labs used the RT configuration. 4
TABLE 5. THE INTRA-ASSAY, INTER-ASSAY AND TOTAL ASSAY STANDARD DEVIATIONS FOR RNA PROFICIENCY PANELS RNA035 (SAMPLES 11-30). DATASETS WITH HIGH INTRA-ASSAY OR INTER-ASSAY SDs ARE LISTED IN BOLD. KIT DATASET INTRA-ASSAY SD INTER-ASSAY SD TOTAL ASSAY SD RC 1 0.06 0.00 0.06 2 0.07 0.00 0.07 3 0.08 0.04 0.09 4 0.11 0.10 0.14 5 0.09 0.10 0.13 6 0.07 0.07 0.10 7 0.07 0.09 0.11 8 0.05 0.07 0.09 RT 9 0.15 0.00 0.15 10 0.16 0.08 0.18 11 0.14 0.08 0.16 12 0.09 0.05 0.11 13 0.16 0.00 0.16 14 0.10 0.00 0.10 15 0.04 0.11 0.12 16 0.14 0.00 0.14 17 0.08 0.11 0.13 18 0.17 0.07 0.19 19 0.14 0.00 0.14 20 0.12 0.00 0.12 21 0.16 0.04 0.17 22 0.18 0.00 0.18 23 0.09 0.00 0.09 24 0.15 0.03 0.15 25 0.15 0.04 0.15 26 0.14 0.00 0.14 27 0.08 0.01 0.08 28 0.13 0.07 0.15 29 0.13 0.05 0.14 30 0.12 0.11 0.16 31 0.16 0.09 0.18 32 0.15 0.00 0.15 33 0.12 0.07 0.14 34 0.14 0.00 0.14 35 0.17 0.07 0.19 36 0.14 0.00 0.14 37 0.14 0.06 0.16 38 0.13 0.11 0.17 39 0.18 0.04 0.18 RT 40 0.10 0.00 0.10 5
KIT DATASET INTRA-ASSAY SD INTER-ASSAY SD TOTAL ASSAY SD ME 41 0.12 0.00 0.12 42 0.05 0.02 0.05 GX 43 0.06 0.07 0.09 AR 44 0.06 0.03 0.07 45 0.07 0.13 0.15 46 0.06 0.07 0.09 47 0.07 0.00 0.07 48 0.04 0.03 0.05 49 0.05 0.06 0.08 50 0.13 0.09 0.16 51 0.04 0.06 0.07 52 0.08 0.12 0.14 53 0.05 0.10 0.11 54 0.04 0.10 0.11 55 0.05 0.03 0.06 56 0.05 0.02 0.05 57 0.04 0.03 0.05 58 0.05 0.03 0.05 59 0.05 0.04 0.06 60 0.05 0.10 0.11 61 0.05 0.00 0.05 62 0.07 0.11 0.13 63 0.08 0.09 0.12 64 0.07 0.10 0.12 65 0.05 0.08 0.09 66 0.08 0.00 0.08 67 0.04 0.00 0.04 68 0.04 0.06 0.07 69 0.08 0.13 0.15 70 0.07 0.04 0.08 71 0.08 0.13 0.15 72 0.08 0.08 0.11 73 0.09 0.00 0.09 74 0.06 0.06 0.09 75 0.07 0.07 0.10 76 0.06 0.04 0.08 77 0.04 0.04 0.06 78 0.12 0.11 0.17 79 0.12 0.07 0.14 80 0.09 0.09 0.13 81 0.08 0.04 0.09 6
KIT DATASET INTRA-ASSAY SD INTER-ASSAY SD TOTAL ASSAY SD AR 82 0.03 0.08 0.08 AR 83 0.06 0.03 0.07 AR 84 0.07 0.09 0.11 AR 85 0.06 0.00 0.06 AR 86 0.06 0.02 0.06 AR 87 0.07 0.10 0.12 AR 88 0.07 0.05 0.09 RC: ROCHE COBAS RT: ROCHE TAQMAN ASSAY; IH: IN-HOUSE ASSAY; GX: GENEXPERT ASSAY; ME: EASYQ STANDARD PREP; AR: ABBOTT REALTIME ASSAY TABLE 6: MEDIAN LOG10 RECOVERY FOR EACH KIT OVER PAST FOUR ROUNDS, RNA035 (SAMPLES 11-30). Excludes samples <100 copies/ml for RT, ME, GX and AR. Assay Number of Labs* Number of Samples Tested* Median Log Recovery ALL 65 979 0.0287 AR 41 510-0.0090 AR-RT 62 891 0.0224 RT 32 381 0.0959 RC*** 4 60 0.14881 * The ALL median log 10 recovery was generated from the data from all laboratories that participated in the four most recent 5-member RNA panels with a commercially available PCR kit; AR: ABBOTT REALTIME; the AR-RT median log 10 recovery was generated from the data from all laboratories that participated in the four most recent 5-member RNA panels with a commercially available real-time PCR kit (specifically, the Abbott RealTime and Roche TaqMan assays); RT: ROCHE TAQMAN. ** Number of labs/samples included in the calculation of the median log recovery value. *** There were 7 labs and 8 datasets for RC. Of these 7 labs, only 4 had the 4 most recent 5-member panels, so the median is based on 4 labs. Median log 10 recovery from Roche Cobas assay for informational purposes only. TABLE 7: THE STANDARD ERROR OF LOG10 RECOVERY OVER PAST FOUR ROUNDS, RNA035 (SAMPLES 11-30), USED IN THE ACCURACY STATISTIC. Excludes samples <100 copies/ml for RT, ME, GX and AR. OBSERVED STANDARD ERROR THRESHOLD STANDARD ERROR STANDARD ERROR USED IN ANALYSIS 0.058201 0.080517 0.080517 7
TABLE 8. ACCURACY SCORING FOR DATASETS GENERATED WITH AN ASSAY THAT DOES NOT HAVE AN ASSAY-SPECIFIC MEDIAN. MEDIAN LOG10 RECOVERY COMPARED TO NOMINAL, A REALTIME MEDIAN AND AN ALL-ASSAY MEDIAN. KIT DATASET Accuracy (Nominal) P-Value Accuracy (RealTime Median) P-Value Accuracy (All-Assay Median) P-Value Accuracy (Kit)* RC 1 2.194 1.915 1.838 0.346 2 1.433 1.155 1.077-0.415 7 1.520 1.242 1.164-0.328 3 1.485 1.207 1.129-0.363 4 1.111 0.833 0.755-0.737 8 1.753 1.475 1.397-0.095 5 0.651 0.373 0.295-1.197 6 1.835 1.557 1.479-0.013 ME 41-0.621-0.899-0.977 ME 42-0.440-0.718-0.796 GX 43 1.366 1.087 1.010 P-Value * Accuracy from Roche Cobas Assay Median for informational purposes only RC: ROCHE COBAS ASSAY; GX: GENEXPERT ASSAY; IH: IN-HOUSE ASSAY; ME: EASYQ STANDARD PREP 8
TABLE 9. PERFORMANCE STATISTICS FOR RNA PROFICIENCY PANELS RNA035 (SAMPLES 11-30) FOR ASSESSMENTS THAT INCLUDE ACCURACY AND THE INTRA-ASSAY STANDARD DEVIATION. (Entries of zero are excluded for clarity) DS: DATASET; VER: ROCHE TAQMAN ASSAY USED (48 OR 96; AmpliPrep or High Pure [HP]) AND VERSION [v2]; INHOUSE ASSAY USED [IH];ABBOTT RT ASSAY USED (0.6ML PROTOCOL [06], AUTOMATED [AUTO] OR MANUAL [MAN] EXTRACTION); MOLECULAR BEACONS EASYQ ASSAY VERSION 2.0 [ME]. NI: THE NUMBER OF INVALID SAMPLES ON THE PANEL; FP: NUMBER OF NEG SAMPLES ABOVE DETECTION LIMIT; FN: NUMBER OF POS SAMPLES BELOW DETECTION LIMIT; NDL: NUMBER OF SAMPLES AT THE LIMIT OF DETECTION; FN50/FN200: NUMBER OF POS SAMPLES AT 50 COPIES/ML BELOW DETECTION LIMIT ON THE ABBOTT RT ASSAY AND THE ROCHE TAQMAN ASSAY/ NUMBER OF POS SAMPLES AT 200 COPIES/ML BELOW DETECTION LIMIT ON THE ME ASSAY; N: THE NUMBER OF SAMPLES INCLUDED IN THE STANDARD DEVIATION; EST: THE TOTAL-ASSAY STANDARD DEVIATION; DIFF: AVERAGE DIFFERENCE BETWEEN THE LOG10 RECOVERY IN A LABORATORY WITH THE EXPECTED LOG10 RECOVERY EXPONENTIATED BACK INTO THE RAW SCALE; ACC: ACCURACY STATISTIC; P: THE P-VALUE FOR EVALUATING FN50/FN200, SD, OR ACCURACY; UNOFF SC: UNOFFICIAL SCORE INCLUDING SCORING FOR SENSITIVITY FOR ABBOTT KITS AND SCORING FOR ACCURACY FOR ROCHE COBAS KITS OFF SC: OFFICIAL SCORE BASED ON PERFORMANCE INCLUDING NON-TECHNICAL SCORING CRITERIA (QUERY RESPONSIVENESS AND INCLUSION OF PHI/PII). THIS EXCLUDES SCORING FOR SENSITIVITY FOR ABBOTT KITS KIT DATASET VER NI NIC FP FN NDL FN50, FN200 TOTAL ASSAY SD ACCURACY N P N EST P DIFF ACC P RC 1 ROCH4800 4 12 0.06 1.204 1.838 C 2 ROCH6800 4 12 0.07 1.046 1.077 C 3 ROCH6800 4 12 0.09 1.056 1.129 C 4 ROCH8800 4 12 0.14 0.985 0.755 C 5 ROCH6800 4 12 0.13 0.905 0.295 C 6 ROCH6800 4 12 0.10 1.127 1.479 C 7 ROCH8800 4 12 0.11 1.063 1.164 C 8 ROCH8800 4 12 0.09 1.110 1.397 C RT 9 RT48v2 4 12 0.15 1.007 0.037 C 10 RT48v2 4 12 0.18 0.965 -.189 C 11 RT48v2 5 15 0.16 1.088 0.454 C 12 RT48v2 4 12 0.11 1.028 0.147 C 13 RT48v2 4 12 0.16 1.148 0.743 C 14 RT48v2 4 12 0.10 1.290 1.375 C 15 RT48v2 4 12 0.12 0.487-3.88 0.05 PC 16 RT48v2 4 12 0.14 1.133 0.676 C 17 RT48v2 4 12 0.13 0.453-4.27 0.01 P 18 RT48v2 4 12 0.19 0.856 -.838 C 19 RT48v2 4 12 0.14 0.943 -.316 C 20 RT48v2 4 12 0.12 0.587-2.87 C 21 RT96v2 4 12 0.17 1.087 0.452 C 22 RT96v2 4 12 0.18 1.015 0.079 C 23 RT96v2 4 12 0.09 1.162 0.809 C 24 RT96v2 5 3 9 0.15 1.138 0.697 P SC 9
KIT DATASET VER NI NIC FP FN NDL FN50, FN200 TOTAL ASSAY SD ACCURACY N P N EST P DIFF ACC P RT 25 RT96v2 4 12 0.15 0.992 -.043 C 26 RT96v2 4 12 0.14 0.988 -.063 C 27 RT96v2 4 12 0.08 1.041 0.218 C 28 RT96v2 4 12 0.15 1.163 0.813 C 29 RT96v2 4 12 0.14 0.844 -.918 C 30 RT96v2 4 12 0.16 1.032 0.169 C 31 RT96v2 4 12 0.18 1.179 0.887 C 32 RT96v2 4 12 0.15 1.056 0.295 C 33 RT96v2 4 12 0.14 0.891 -.621 C 34 RT96v2 4 12 0.14 0.864 -.787 C 35 RT96v2 4 12 0.19 0.921 -.442 C 36 RT96v2 4 12 0.14 0.796-1.23 C 37 RT96v2 4 12 0.16 1.064 0.333 C 38 RT96v2 4 12 0.17 0.845 -.910 C 39 RT96v2 4 12 0.18 1.061 0.318 C 40 RT96v2 4 12 0.10 1.125 0.636 C ME 41 STDv1 4 12 0.12 0.910 -.621 C 42 STDv1 4 12 0.05 0.941 -.440 C GX 43 GeneXper 4 12 0.09 1.033 1.010 C AR 44 06AUTO 4 12 0.07 0.832 -.994 C 45 06AUTO 1 4 1 12 0.15 0.767-1.43 C 46 06AUTO 4 12 0.09 1.075 0.390 C 47 06AUTO 4 12 0.07 0.953 -.258 C 48 06AUTO 4 12 0.05 1.001 0.007 C 49 06AUTO 4 12 0.08 0.956 -.244 C 50 06MAN 4 12 0.16 0.968 -.176 C 51 06AUTO 4 12 0.07 1.252 1.212 C 52 06AUTO 4 12 0.14 1.150 0.756 C 53 06AUTO 4 12 0.11 1.082 0.424 C 54 06AUTO 1 4 12 0.11 1.032 0.171 PC 55 06AUTO 4 12 0.06 1.174 0.867 C 56 06MAN 4 12 0.05 1.211 1.035 C 57 06AUTO 4 12 0.05 0.980 -.107 C 58 06AUTO 4 12 0.05 1.100 0.516 C 59 06AUTO 4 12 0.06 0.863 -.795 C 60 06MAN 4 12 0.11 1.209 1.022 C 61 06MAN 4 12 0.05 1.049 0.257 C 62 06AUTO 4 12 0.13 1.131 0.665 C 63 06AUTO 4 12 0.12 0.901 -.562 C 64 06MAN 4 12 0.12 0.873 -.734 C 65 06AUTO 4 12 0.09 0.996 -.021 C SC 10
KIT DATASET VER NI NIC FP FN NDL FN50, FN200 TOTAL ASSAY SD ACCURACY N P N EST P DIFF ACC P AR 66 06AUTO 4 12 0.08 0.953 -.259 C 67 06AUTO 4 12 0.04 0.922 -.439 C 68 06AUTO 4 12 0.07 1.076 0.396 C 69 06AUTO 4 12 0.15 1.316 1.480 C 70 06AUTO 4 12 0.08 1.077 0.402 C 71 06AUTO 4 12 0.15 1.031 0.165 C 72 06AUTO 4 12 0.11 1.018 0.097 C 73 06AUTO 4 12 0.09 0.997 -.014 C 74 06AUTO 4 12 0.09 1.227 1.103 C 75 06MAN 5 3 9 0.10 1.275 1.308 P 76 06AUTO 4 12 0.08 0.911 -.500 C 77 06AUTO 4 12 0.06 0.978 -.119 C 78 06AUTO 4 12 0.17 1.104 0.534 C 79 06MAN 4 12 0.14 0.926 -.415 C 80 06MAN 4 12 0.13 1.179 0.887 C 81 06AUTO 4 12 0.09 0.953 -.260 C 82 06MAN 4 12 0.08 1.002 0.009 C 83 06AUTO 4 12 0.07 0.881 -.686 C 84 06AUTO 4 12 0.11 0.837 -.961 C 85 06AUTO 4 12 0.06 0.879 -.699 C 86 06AUTO 4 12 0.06 0.851 -.873 C 87 06AUTO 4 12 0.12 1.001 0.006 C 88 06AUTO 4 12 0.09 1.102 0.525 C SC RC: ROCHE COBAS ASSAY; RT: ROCHE TAQMAN ASSAY; IH: IN-HOUSE ASSAY; ME: EASYQ STANDARD PREP; AR: ABBOTT ASSAY DATA SETS WITH PROBLEMS ARE HIGHLIGHTED WITH SHADING. 11
TABLE 10. COMPARISON OF KIT LOT NUMBER, MEAN LOG10 RECOVERY, TOTAL ASSAY SD, AND SCORE FOR DATA GENERATED WITH THE ROCHE TAQMAN ASSAY IN RNA035 (SAMPLES 11-30). SHADED ROWS REPRESENT DATASETS WITH TECHNICAL PROBLEMS IN THIS LATEST ROUND OF TESTING. RED FONT INDICATES LABORATORIES THAT ARE BEING MONITORED BUT DID NOT SUBMIT DATA AND GET SCORED FOR THIS ROUND. VER DATASET KIT LOT MEAN LOG 11 TOTAL ASSAY SD SCORE 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 RT48v2 9 Y24080 E00164 E00164 E11392 0.06 0.17 0.09 0.07 0.09 0.10 0.12 0.15 C C C C RT48v2 10 E00164 E00164 E00164 E00164-0.06 0.16 0.02 0.21 0.16 0.17 0.17 0.18 C C C C RT48v2 - E00164 EXE OH OH -0.03 EXE OH OH 0.15 EXE OH OH C EXE OH OH RT48v2 11 EXE EXE EXE Y17461 EXE EXE EXE 0.25 EXE EXE EXE 0.16 EXE EXE EXE C RT48v2 12 Y24080 E00164 E00164 E11392 0.01 0.17 0.10 0.16 0.13 0.12 0.11 0.11 C C C C RT48v2 13 Y18458 Y14818 Y22924 Y28679-0.07 0.21 0.13 0.12 0.22 0.11 0.13 0.16 PC C C C RT48v2 - Y24080 OH OH OH 0.09 OH OH OH 0.13 OH OH OH C OH OH OH RT48v2 14 Y14818 Y21188 Y14818 Y21188 0.17 0.21 0.17 0.28 0.11 0.10 0.10 0.10 C C C C RT48v2HP 15 - - - - -0.07-0.28-0.20-0.31 0.19 0.19 0.18 0.12 P P P PC RT48v2 - Y24080 Y24080 Y14818 EXT 0.03 0.27 0.09 EXT 0.13 0.17 0.18 EXT C C C EXT RT96v2 - OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH OH RT48v2 16 X09013 Y17461 Y17461 Y17461 0.09 0.21 0.14 0.16 0.13 0.13 0.14 0.14 C C C C RT48v2HP 17 - - - -0.34-0.34-0.09-0.22 0.24 0.22 0.17 0.13 P P PC P RT48v2 18 Y15616 E00164 E00164 E11392-0.10 0.15-0.05 0.11 0.18 0.17 0.17 0.19 C C C C RT48v2 19 Y24080 E00164 E00164 E11392 0.02 0.09 0.05 0.12 0.15 0.14 0.14 0.14 C C C C RT48v2HP 20 - - -0.19-0.08-0.08-0.19 0.09 0.11 0.12 0.12 PC PC PC C RT48v2 21 Y24080 Y24080 E00164 E11392 0.00 0.21 0.21 0.11 0.14 0.14 0.16 0.17 C C C C RT96v2 - EXE OH OH OH EXE OH OH OH EXE OH OH OH EXE OH OH OH RT96v2 22 Y18458 Y21188 Y21188 Y27450 0.04 0.21 0.09 0.08 0.18 0.18 0.16 0.18 C C C C RT96v2 23 X14181 Y16242 Y16242 Y16242 0.18 0.21 0.15 0.11 0.16 0.13 0.11 0.09 C C C C 12
VER DATASET KIT LOT MEAN LOG 10 TOTAL ASSAY SD SCORE 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 RT96v2 24 Y10651 Y10651 Y16242 Y23998 0.07 0.25-0.13 0.14 0.13-0.15 C C P P RT96v2 25 Y17461 Y21188 Y28679 Y28679 0.03 0.24 0.05 0.05 0.17 0.15 0.17 0.15 C C C C RT96v2 26 Y18458 Y18458 Y25278 Y25278 0.06 0.07 0.02 0.20 0.15 0.13 0.14 0.14 C C C C RT96v2 27 Y10651 Y10651 Y10651 Y16242 0.09 0.14 0.06 0.16 0.09 0.09 0.09 0.08 C C C C RT96v2 28 Y11397 Y11397 Y17121 Y17121 0.03-1.29 0.13 0.27 0.16 1.03 0.13 0.15 C P C C RT96v2 29 Y28460 Y23998 E00588 E05473-0.01 0.16-0.01-0.05 0.15 0.12 0.15 0.14 C C C C RT96v2 30 Y14818 Y21188 E00588 Y23998 0.15 0.26-0.05 0.08 0.12 0.12 0.16 0.16 C C C C RT96v2 31 Y03434 Y17121 Y17121 Y26634 0.26 0.29 0.04 0.08 0.16 0.12 0.16 0.18 C C C C RT96v2 32 Y16436 Y17461 Y25278 E00588 0.02 0.26 0.11 0.18 0.18 0.21 0.18 0.15 C PC C C RT96v2 33 X09013 Y2399800 Y2399800 Y23998 0.09 0.12-0.056 0.07 0.12 0.12 0.15 0.14 C C C C RT96v2 34 Y18458 Y22924 E00588 Y28679 0.00 0.02 0.02 0.09 0.17 0.15 0.15 0.14 C C C C RT96v2 - Y20212 E00588 Y25278 EXT 0.05 0.17-0.01 EXT 0.15 0.15 0.16 EXT C C C EXT RT48v2 35 Y16436 Y16436 Y18458-0.06 0.22 0.00 0.09 0.19 0.19 0.20 0.19 C C C C RT96v2 36 Y17121 Y20212 Y20212 Y20212-0.03 0.06-0.10 0.05 0.17 0.17 0.15 0.14 C C C C RT96v2 37 Y10651 Y08928 Y25278 Y25278 0.03 0.26 0.05 0.14 0.14 0.16 0.16 0.16 C C C C RT96v2 38 Y14632 Y21188 Y22924 Y27450-0.25 0.12 0.01-0.16 0.21 0.11 0.13 0.17 PC C C C RT96v2 - X13007 Y11397 Y113A EXT 0.08 0.14 0.25 EXT 0.15 0.17 0.19 EXT C C C EXT RT96v2 39 Y17461 Y22924 E00588 Y25278 0.13 0.22-0.03 0.16 0.15 0.14 0.08 0.18 C C C C RT96v2 40 E00164 E00164 E00164 E00164 0.12 0.17 0.14 0.15 0.08 0.06 0.08 0.10 C C C C 13
TABLE 11. COMPARISON OF KIT LOT NUMBER, MEAN LOG10 RECOVERY, TOTAL ASSAY SD, AND SCORE FOR DATA GENERATED WITH THE ABBOTT REALTIME ASSAY IN RNA035 (SAMPLES 11-30). SHADED ROWS REPRESENT DATASETS WITH TECHNICAL PROBLEMS IN THIS LATEST ROUND OF TESTING. RED FONT INDICATES LABORATORIES THAT ARE BEING MONITORED BUT DID NOT SUBMIT DATA AND GET SCORED FOR THIS ROUND. VER DATASET KIT LOT MEAN LOG 11 TOTAL ASSAY SD SCORE 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-5 AR0.6AUT 44 480594 480538 480538 487561-0.04-0.13-0.12-0.07 0.10 0.06 0.06 0.07 C C C C AR0.6AUT 45 480594 480594 487982 487982-0.32-0.13-0.03-0.02 0.16 0.13 0.14 0.15 C C C C AR0.6AUT 46 482420 485089 482420 488464 0.01-0.05 0.13 0.00 0.11 0.12 0.12 0.09 C C C C AR0.6AUT 47 481998 481998 483548 487985-0.06-0.01-0.02-0.03 0.11 0.12 0.12 0.07 C C C C AR0.6AUT 48 478983 478065 478065 485317 0.03-0.06-0.02 0.01 0.05 0.07 0.07 0.05 C C C C AR0.6AUT 49 480461 480461 480461 485089 0.06-0.09-0.01-0.07 0.06 0.09 0.09 0.08 C C C C AR0.6MAN 50 481685 481685 487988 487988-0.03-0.18 0.10 0.02 0.18 0.17 0.20 0.16 C C C C AR0.6AUT 51 480460 479390 483847 487985 0.16 0.10 0.01 0.08 0.05 0.05 0.07 0.07 C C C C AR0.6AUT 52 480460 481512 481998 485079 0.08-0.13 0.16 0.10 0.10 0.16 0.17 0.14 C C C C AR0.6AUT 53 478983 482609 483361 484888-0.11 0.12-0.01 0.10 0.12 0.13 0.11 0.11 C C C C AR0.6AUT 54 478984 478984 485081 485081 0.06-0.15 0.03 0.08 0.08 0.12 0.11 0.11 C PC PC PC AR0.6AUT 55 480460 482418 483974 485081 0.04 0.12 0.06 0.03 0.08 0.09 0.09 0.06 C C C C AR0.6MAN 56 480460 482418 483974 485081-0.17 0.04-0.02 0.08 0.11 0.07 0.11 0.05 C C C C AR0.6AUT 57 485508 487561 487561 487982-0.05 0.04-0.04-0.02 0.13 0.11 0.05 0.05 C C C C AR0.6AUT 58 478845 478845 478845 487571 0.08 0.04 0.03-0.02 0.04 0.04 0.05 0.05 C C C C AR0.6AUT 59 478665 483864 485081 485081 0.00-0.09-0.10-0.11 0.13 0.07 0.08 0.06 C C C C AR0.6MAN 60 480463 484888 486856 486856 0.22 0.06 0.03-0.02 0.10 0.12 0.12 0.11 C C C C AR0.6MAN 61 481996 485089 485089 485795 0.00 0.04 0.02-0.02 0.05 0.05 0.05 0.05 C C C C AR0.6AUT 62 480461 480945 480945 485508-0.06 0.00 0.03 0.21 0.07 0.08 0.08 0.13 C C C C AR0.6AUT 63 483513 483513 483513 483513 0.09-0.07-0.15-0.08 0.05 0.08 0.12 0.12 C C C C AR0.6MAN 64 483513 483513 483513 483513-0.10-0.03-0.03-0.14 0.09 0.06 0.08 0.12 C C C C AR0.6AUT - 480460 479390 OH OH -0.05-0.07 OH OH 0.04 0.06 OH OH C C OH OH 14
KIT LOT MEAN LOG 11 TOTAL ASSAY SD SCORE VER DATASET 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 AR0.6AUT 65 480460 479390 481998 485079 0.03 0.08-0.05-0.10 0.13 0.08 0.08 0.09 C C C C AR0.6AUT 66 480460 481512 483513 483847-0.01-0.07 0.01-0.06 0.06 0.08 0.10 0.08 C C C C AR0.6AUT 67 480460 481512 482739 483847-0.05-0.02-0.07-0.03 0.07 0.07 0.06 0.04 C C C C AR0.6AUT 68 480460 479390 481512 483847 0.05 0.09 0.00-0.05 0.11 0.12 0.05 0.07 C C C C AR0.6AUT 69 480594 482912 486849 488964 0.14 0.18 0.21-0.09 0.11 0.05 0.07 0.15 C C C C AR0.6AUT 70 481140 478983 484888 486856 0.06 0.26 0.00-0.04 0.06 0.05 0.07 0.08 C C C C AR0.6AUT 71 483864 483864 483864 487351-0.02 0.15 0.06-0.17 0.14 0.13 0.09 0.15 C C C C AR0.6AUT 72 474399 485081 485081 485081-0.12 0.09 0.06-0.03 0.07 0.10 0.11 0.11 C C C C AR0.6AUT 73 476172 476172 487571 487571 0.02-0.02-0.06 0.02 0.09 0.06 0.08 0.09 C C C C AR0.6AUT 74 EXE EXE 482573 486731 EXE EXE 0.02 0.03 EXE EXE 0.09 0.09 EXE EXE C C AR0.6MAN 75 EXE EXE 482573 486731 EXE EXE 0.24 - EXE EXE 0.11 - EXE EXE C P AR0.6AUT 76 480460 479390 482739 483847-0.08 0.03-0.07-0.08 0.16 0.17 0.08 0.08 C C C C AR0.6AUT 77 479626 482609 483361 486856-0.05-0.02 0.05-0.05 0.06 0.06 0.07 0.06 C C C C AR0.6AUT 78 480463 478984 480463 487352 0.02-0.14 0.06 0.19 0.07 0.11 0.13 0.17 C C C C AR0.6MAN 79 480463 478984 480463 487352 0.11-0.13-0.03-0.11 0.06 0.11 0.11 0.14 C C C C AR0.6MAN 80 479047 479047 479047 486731 0.22 0.02-0.01 0.01 0.11 0.11 0.13 0.13 C C C C AR0.6AUT 81 479390 481512 481998 483847-0.11-0.02 0.02 0.00 0.12 0.09 0.08 0.09 C C C C AR0.6MAN 82 480461M 480461M 486849 486849 0.05 0.05-0.01-0.12 0.05 0.04 0.04 0.08 C C C C AR0.6AUT 83 483864 483864 483864 483864-0.06-0.03-0.04-0.13 0.10 0.08 0.05 0.07 C C C C AR0.6AUT 84 480594 486730 486730 486730-0.18-0.01 0.01-0.16 0.16 0.16 0.16 0.11 C C C C AR0.6AUT 85 479850 479850 479850 479850-0.06-0.07-0.06-0.07 0.12 0.11 0.09 0.06 C C C C AR0.6AUT 86 481996 481996 485795 486730-0.09-0.10-0.02-0.10 0.13 0.15 0.13 0.06 C C C C AR0.6AUT 87 480460 481512 482739 486731 0.05-0.14 0.10-0.04 0.05 0.09 0.11 0.12 C C C C AR0.6AUT 88 480460 481512 483847 487985-0.37-0.01 0.02 0.13 0.20 0.10 0.08 0.09 PC C C C AR0.6AUT - 481490 476172 481490 EXT 0.12-0.16 0.40 EXT 0.20 0.18 0.23 EXT C PC P EXT AR0.6MAN - 478665 485509 483974 EXE 0.05-0.08-0.07 EXE 0.07 0.07 0.07 EXE C C C EXE 15
TABLE 12. COMPARISON OF KIT LOT NUMBER, MEAN LOG10 RECOVERY, TOTAL ASSAY SD, AND SCORE FOR DATA GENERATED WITH THE ROCHE COBAS ASSAY IN RNA035 (SAMPLES 11-30). SHADED ROWS REPRESENT DATASETS WITH TECHNICAL PROBLEMS IN THIS LATEST ROUND OF TESTING. RED FONT INDICATES LABORATORIES THAT ARE BEING MONITORED BUT DID NOT SUBMIT DATA AND GET SCORED FOR THIS ROUND. VER DATASET KIT LOT MEAN LOG 11 TOTAL ASSAY SD SCORE 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 ROCH4800 1 - - - - - 0.17 0.20 0.17 0.09 0.09 0.09 0.06 P C C C ROCH6800 2 - - - - 0.06 0.12 0.15 0.14 0.08 0.07 0.07 0.07 C C C C ROCH6800 3 - - - - 0.07 0.09 0.12 0.10 0.10 0.09 C C C ROCH6800 4 - - - - 0.08-0.07 0.19 0.02 0.17 0.12 0.14 0.13 C C C C ROCH6800 5 - - - - 0.23 0.16 0.16 0.03 0.10 0.09 0.08 0.10 C C C C ROCH8800 6 - - - - 0.17 0.17 0.18 0.13 0.10 0.11 C C C ROCH8800 7 - - - - 0.12 0.23 0.14 0.11 0.13 0.14 C C C ROCH8800 8 - - - - 0.23 0.05 0.16 0.10 0.12 0.13 0.11 0.09 C C C C 16
TABLE 13. COMPARISON OF KIT LOT NUMBER, MEAN LOG10 RECOVERY, TOTAL ASSAY SD, AND SCORE FOR DATA GENERATED WITH THE GENEXPERT ASSAY IN RNA035 (SAMPLES 11-30). SHADED ROWS REPRESENT DATASETS WITH TECHNICAL PROBLEMS IN THIS LATEST ROUND OF TESTING. RED FONT INDICATES LABORATORIES THAT ARE BEING MONITORED BUT DID NOT SUBMIT DATA AND GET SCORED FOR THIS ROUND. VER DATASET KIT LOT MEAN LOG 11 TOTAL ASSAY SD SCORE 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 35-3 35-4 35-5 35-6 GeneXpert - - - - EXT 0.16 0.17 0.17 EXT 0.20 0.10 0.08 EXT C C C EXT GeneXpert - - - - EXT 0.13 0.08 0.23 EXT 0.05 0.05 0.09 EXT C C C EXT GeneXpert - - - - EXT 0.15 0.24 0.20 EXT 0.04 0.08 0.09 EXT C C C EXT GeneXpert - - - - EXT 0.24 0.09 0.06 EXT 0.10 0.09 0.10 EXT C C C EXT GeneXpert - - - - - 0.17 0.16 0.05 0.05 C C GeneXpert - - - - EXT 0.16 0.18 0.11 EXT 0.17 0.18 0.18 EXT C C C EXT GeneXpert - - - - EXT 0.12 0.17 0.07 EXT 0.07 0.08 0.08 EXT C C C EXT GeneXpert - - - - EXT 0.12-0.06-0.30 EXT 0.07 0.10 0.19 EXT C C C EXT GeneXpert - - - - - 0.04 0.04 C GeneXpert 43 - - - - 0.15 0.19 0.04 0.06 0.04 0.04 0.09 0.09 C C C C GeneXpert - - - - EXT 0.05 0.10 EXT 0.06 0.05 EXT C C EXT 17
Mean LOG 10 RNA Recovery Mean LOG 10 RNA Recovery VIROLOGY QUALITY ASSURANCE PROGRAM FIGURE 1. LINE PLOT OF MEAN LOG10 RECOVERY FOR DATA GENERATED WITH THE ROCHE TAQMAN ASSAY IN RNA028 SAMPLES, RNA029 SAMPLES, RNA030 SAMPLES, RNA031 SAMPLES, RNA032 SAMPLES, RNA033 SAMPLES AND RNA035 SAMPLES 1-30. LABORATORIES WITHOUT ( left panel ) and WITH (right panel) SCORING PROBLEMS. Taqman: Labs with all C scores Taqman: Labs with at least one PC or P score 0.4 Kits S02574 and higher High Pure Kits Adjusted Means Other Kits 0.4 Kits S02574 and higher Adjusted Means value= 0.467 High Pure Kits Other Kits Adjusted Means without HP 0.2 0.2 0.0 0.0-0.2-0.2-0.4-0.4 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-30 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-30 18
Mean LOG 10 RNA Recovery Mean LOG 10 RNA Recovery VIROLOGY QUALITY ASSURANCE PROGRAM FIGURE 2. LINE PLOT OF MEAN LOG10 RECOVERY FOR DATA GENERATED WITH THE ABBOTT ASSAY IN RNA028 SAMPLES, RNA029 SAMPLES, RNA030 SAMPLES, RNA031 SAMPLES, RNA032 SAMPLES, RNA033 SAMPLES AND RNA035 SAMPLES 1-30. LABORATORIES WITHOUT (left panel) and WITH (right panel) SCORING PROBLEMS. 0.4 Abbott: Labs with all C scores Adjusted Means 0.4 Abbott: Labs with at least one PC or P score value= 0.465 Adjusted Means value= 0.416 0.2 0.2 0.0 0.0-0.2-0.2-0.4-0.4 value= -0.896 value= -1.497 value= -0.697 value= -0.925 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-30 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-30 19
Mean LOG 10 RNA Recovery Mean LOG 10 RNA Recovery VIROLOGY QUALITY ASSURANCE PROGRAM FIGURE 3. LINE PLOT OF MEAN LOG10 RECOVERY FOR DATA GENERATED WITH THE GENEXPERT (left panel) and ROCHE COBAS (right panel) ASSAYs IN RNA029 SAMPLES, RNA030 SAMPLES, RNA031 SAMPLES, RNA032 SAMPLES, RNA033 SAMPLES AND RNA035 SAMPLES 1-30. ALL LABORATORIES INCLUDED. GeneXpert: All Labs Roche Cobas: All Labs 0.4 Adjusted Means 0.4 Adjusted Means 0.2 0.2 0.0 0.0-0.2-0.2-0.4-0.4 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-25 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-30 20