TO: CC: Members of the VQA Advisory Board (VQAAB) Bill Meyer Bob Coombs/Ming Chang Nicole Tobin Belinda Yen-Lieberman Joan Dragavon Urvi Parikh Jessica Fogel James Bremer Cheryl Jennings Carolyn Yanavich/Diane Costello Anita Gillis/Jenny Nguyen Heather Sprenger Ron Bosch Joe Fitzgibbon, Keith Crawford FROM: Don Brambilla, Dan Zaccaro, Amy Couzens and Barbara Do DATE: SUBJECT: Samples 21-25 of RNA035, Revised The following is a revision of the first report on the analysis of VQA HIV RNA proficiency testing data through samples 21-25 of RNA035. In this revision, we have corrected the description of the results for dataset 93, and removed one dataset (dataset 37) from the analysis. The removed dataset has been re-analyzed and included in the secondary analysis of samples 21-25 of RNA035. Ninety-three datasets from 70 laboratories were included in the analysis. Datasets that were generated by different kits in the same laboratory were analyzed separately. Eighty-seven of the 93 datasets consisted of results from the most recent four 5-member panels (6-25 of RNA035). Datasets 3 and 50 consisted of the results of a 20- member qualifying panel that was recently tested plus 1-3 5-member panels that were tested after the 20-member panel. As usual, the composition of the panels depended upon the kit that was used in a given laboratory (Tables 1-4). Proficiency panels are scored on the total assay standard deviation of log10 recovery, accuracy, counts of negative results near the limit of detection, false positives and invalid results. The total assay standard deviation (SD) is related to the intra-assay and inter-assay standard deviations as ST=(SA 2 +SE 2 ) 1/2 where ST is the total assay SD, SA is the intra-assay SD and SE is the inter-assay SD. The intra-assay, inter-assay and total assay SD are calculated for each data set (Table 5) and are used to identify which data sets exhibit problems with precision and which component of variation is contributing to the elevated precision statistic. Sensitivity is assessed by determining the number of negative results obtained from replicates at the limit of detection for the assay in question. The limit of detection is defined as the nominal concentration at which 95% of assay results are expected to be positive (Roche COBAS AmpliPrep / COBAS TaqMan HIV-1, version 2 assay (Roche TaqMan), Abbott RealTime HIV-1 assay (Abbott RealTime), and internally developed taqman (IH) assays: 50 copies/ml; and biomerieux NucliSens EasyQ HIV RNA (EasyQ): 200 copies/ml). Specificity is assessed by monitoring the results obtained on samples that do not contain HIV. It is also expected that all results should meet established criteria for assay validity. Accuracy is assessed by comparing average(mean) log10 recovery in a laboratory with expected log10 recovery. The average is calculated using the subset of samples that is used to assess the total assay standard deviation. Median log10 recovery across all laboratories in which the same kit was used on the panels in question is treated as the expected value. Estimated kit-specific log10 values are calculated for each analysis (Table 6). Per the VQAAB s recommendation, an analysis investigating the impact of standard errors on the accuracy statistics on previous rounds of testing was conducted in April 2009. As a result of this investigation, a modified approach for the scoring procedure is implemented when the SE on a round of testing is low enough to be of concern. Under this modified approach, the observed standard 1
error on a round of testing is replaced by the historical median of 0.080517 when the observed SE is below the median; otherwise, the observed value would continue to be used (Table 7). A minimum of 7 data sets must be available in order to generate a reliable median. If fewer than 7 data sets are available for any given assay, then accuracy is assessed using modified criteria. In this case, median log10 recovery is compared to 1) the nominal concentration; 2) an overall real-time cross-platform derived median; and 3) an overall cross-platform derived median (Table 8). The overall real-time cross-platform-derived median was determined using data only from all the commercially available real-time PCR kits (specifically, Abbott RealTime and Roche TaqMan). The overall crossplatform-derived median is determined using data from all the commercially available PCR kits. All three medians are used for determining a final accuracy score. If the same penalty is assessed for 2 of the 3 medians, then that penalty is assessed (this could be a no-penalty scenario in which 2 out of the 3 p-values are > 0.05). If the results indicated three different penalties (i.e. >0.05, between 0.05 and 0.01, or <0.01) then a penalty of between 0.05 and 0.01 is assessed, which is equivalent to a PC. In this analysis, there were fewer than 7 data sets available for the IT and EasyQ assays. The modified criteria for assessing accuracy were used to evaluate accuracy in these data sets. The VQA will now be providing official scores for non-technical criteria including the timeliness of query responsiveness and the submission of data containing patient identifiable information or protected health information (PII/PHI). The inclusion of these non-technical criteria were unofficial for two rounds of testing starting with the scoring of RNA032-1 (July 2016). The official scoring of these non-technical criteria will begin with the scoring of samples including RNA032-3 (November 2016). The non-technical scoring nomenclature will help to differentiate between problems with late data submissions (T), unresponsiveness to data queries (Q) and submission of data containing PII/PHI (P). A number of 1 or 2 will be used to indicate if one (1) or more (2) of a given problem was noted. Multiple letters will indicate if multiple problems were noted in the last four analyses (C1 T,P or C2 Q ). Technical scores will be indicated as done previously (C, PC, P) and the nontechnical problems will be indicated as superscripts to the technical score. The numeric score for each round that is used for determining the cumulative rating will be based on the combined technical and non-technical criteria. Beginning in September 2017, the VQA provided both an official and an unofficial score based on reported Abbott false negative results derived from the 50cp/mL controls. The official score did not incorporate sensitivity results based on 50cp/mL samples; unofficial scores reflected false negative results as per our historical criteria. Sensitivity results were excluded from official scores due to an investigation into the increased rate of false negative results (from Abbott kits) for VQA HIV RNA proficiency testing samples with a nominal value of 50 cp/ml. As of the primary analysis for RNA panel 035, samples 6-10, the VQA has resumed official scoring for sensitivity for Abbott data starting with samples 6-10 of RNA035. Any false negatives reported for panels RNA032 or RNA033 will not be included in sensitivity scoring, and have been removed from Table 9; only data from RNA035 testing or later will be used for sensitivity scoring. Beginning in March 2018, the VQA will be providing both an official and an unofficial score based on accuracy results for the Roche cobas HIV-1 quantitative nucleic acid test. The official score will not incorporate accuracy statistics; unofficial scores will reflect accuracy statistics per the historical criteria. Accuracy results will be excluded from official scores until sufficient numbers of data sets (>7) are available to permit the estimation of within-assay median values for use in accuracy scoring. It is anticipated that this report will be discussed during our next conference call. We can be reached for questions or comments via email (dzaccaro@rti.org), telephone (919-541-6310) or FAX (919-541-5966). 2
RESULTS: ROCHE COBAS ASSAY No technical problems were identified in the 8 datasets generated with the Roche cobas HIV-1 assay (Table 9). Scores of C are recommended for the datasets generated with this assay. Trends in log10 recovery with Roche Cobas kit data are illustrated in Table 12 and Figure 3. RESULTS: ROCHE TAQMAN ASSAY Technical problems were identified in 5 of the 30 datasets generated with the Roche TaqMan assay (Table 9). Accuracy issues were noted in datasets 14, 16 and 19. Scores of PC are recommended for datasets 16 and 19 (accuracy statistics of -3.78 and -3.05 were observed), and a score of P is recommended for dataset 14 (an accuracy statistic of -4.42 was observed). A failed VQA 200 copy control (VQA200 reported as 708 copies/ml) was observed in dataset 23. This invalidates all 5 samples. A score of P is recommended for dataset 23. Scores of C are recommended for the other 25 datasets generated with this assay. We continue to monitor log10 recovery for laboratories using the Roche TaqMan Assay (Table 10; Figure 1) and will continue to share data with Roche Molecular Systems (RMS) on the performance of their assay; new analyses looking specifically at log10 recovery between data sets generated with AmpliPrep vs. High Pure manual extractions have been added to this monitoring. RESULTS: IN-HOUSE ASSAY No technical problems were identified in the 1 dataset generated with the In-house (IH) assay (Table 9). A score of C is recommended for the dataset generated with this assay. RESULTS: EASYQ ASSAY No technical problems were identified in the 2 datasets generated with the EASYQ assay (Table 9). Scores of C are recommended for the datasets generated with this assay. RESULTS: GENEXPERT ASSAY No technical problems were identified in the 8 datasets generated with the GeneXpert assay (Table 9). Scores of C are recommended for the datasets generated with this assay. Trends in log10 recovery with GeneXpert kit data are illustrated in Table 13 and Figure 3. RESULTS: ABBOTT REALTIME ASSAY Technical problems were identified in 2 of the 44 datasets generated with the Abbott RealTime assay (Table 9). One false positive result was noted in dataset 61 which was a carryover from the last round (35-4), but the laboratory did not submit a repeat panel. A score of PC is recommended for this dataset. Two false negative results and an elevated total assay standard deviation due to increased inter-assay standard deviation were noted in dataset 93. Both of these false negatives are from RNA proficiency panel 035 (one from round 35-3 and one from 35-4, but the laboratory did not submit a repeat panel). A score of P is recommended for dataset 93. Scores of C are recommended for the other 42 datasets generated with this assay. Trends in log10 recovery with Abbott kit data are illustrated in Table 11 and Figure 2. The Abbott mean log10 recovery continues to be relatively flat. No problems with query unresponsiveness or the submission of data that included patient identifiable information (PII) or protected health information (PHI) were noted in any of the data sets included in this analysis. It is anticipated that this report will be discussed during our next conference call. We can be reached for questions or comments via email (dzaccaro@rti.org), telephone (919-541-6310) or FAX (919-541-5966). 3
TABLE 1. COMPOSITION OF SAMPLES 6-10 OF RNA PROFICIENCY PANEL RNA035*. NOMINAL CONCENTRATION (COPIES/ML) RT 0 1 50 1 850 1 4,250 1 34,000 238,000 1 TABLE 2. COMPOSITION OF SAMPLES 11-15 OF RNA PROFICIENCY PANEL RNA035*. NOMINAL CONCENTRATION (COPIES/ML) RT 0 1 50 1 850 1 4,250 34,000 1 238,000 1 TABLE 3. COMPOSITION OF SAMPLES 16-20 OF RNA PROFICIENCY PANEL RNA035*. NOMINAL CONCENTRATION (COPIES/ML) RT 0 1 50 1 850 1 4,250 1 34,000 1 238,000 TABLE 4. COMPOSITION OF SAMPLES 21-25 OF RNA PROFICIENCY PANEL RNA035*. NOMINAL CONCENTRATION (COPIES/ML) RT 0 1 50 1 850 1 4,250 34,000 1 238,000 1 *Prior to panel RNA035, the OT configuration was used by labs using the EasyQ assay; the RT configuration was used by labs testing the Roche Cobas, TaqMan, In-House, GeneXpert or Abbott RealTime assays. Starting in panel RNA035, all labs used the RT configuration. 4
TABLE 5. THE INTRA-ASSAY, INTER-ASSAY AND TOTAL ASSAY STANDARD DEVIATIONS FOR RNA PROFICIENCY PANELS RNA035 (SAMPLES 6-25). DATASETS WITH HIGH INTRA-ASSAY OR INTER-ASSAY SDs ARE LISTED IN BOLD. KIT DATASET INTRA-ASSAY SD INTER-ASSAY SD TOTAL ASSAY SD RC 1 0.09 0.00 0.09 2 0.06 0.04 0.07 3 0.07 0.08 0.10 4 0.11 0.09 0.14 5 0.06 0.06 0.08 6 0.05 0.09 0.10 7 0.08 0.10 0.13 8 0.06 0.08 0.11 RT 9 0.12 0.00 0.12 10 0.17 0.00 0.17 11 0.09 0.06 0.11 12 0.13 0.00 0.13 13 0.10 0.00 0.10 14 0.04 0.17 0.18 15 0.14 0.11 0.18 16 0.10 0.14 0.17 17 0.16 0.06 0.17 18 0.14 0.00 0.14 19 0.11 0.05 0.12 20 0.15 0.05 0.16 21 0.16 0.00 0.16 22 0.11 0.00 0.11 23 0.12 0.06 0.13 24 0.16 0.06 0.17 25 0.14 0.00 0.14 26 0.09 0.00 0.09 27 0.13 0.03 0.13 28 0.14 0.06 0.15 29 0.08 0.14 0.16 30 0.15 0.09 0.18 31 0.15 0.00 0.15 32 0.16 0.00 0.16 33 0.19 0.05 0.20 34 0.14 0.04 0.15 35 0.15 0.06 0.16 36 0.13 0.01 0.13 38 0.17 0.05 0.18 39 0.08 0.00 0.08 5
KIT DATASET INTRA-ASSAY SD INTER-ASSAY SD TOTAL ASSAY SD IH 40 0.05 0.06 0.08 ME 41 0.12 0.00 0.12 42 0.06 0.04 0.07 GX 43 0.08 0.00 0.08 44 0.06 0.06 0.09 45 0.08 0.03 0.09 46 0.07 0.07 0.10 47 0.13 0.13 0.18 48 0.05 0.05 0.08 49 0.05 0.19 0.19 50 0.05 0.00 0.05 AR 51 0.04 0.04 0.06 52 0.08 0.11 0.14 53 0.06 0.10 0.12 54 0.07 0.09 0.12 55 0.04 0.05 0.07 56 0.04 0.08 0.09 57 0.18 0.07 0.20 58 0.04 0.06 0.07 59 0.09 0.14 0.17 60 0.06 0.08 0.11 61 0.03 0.11 0.11 62 0.05 0.07 0.09 63 0.10 0.05 0.11 64 0.03 0.04 0.05 65 0.05 0.00 0.05 66 0.05 0.07 0.08 67 0.02 0.12 0.12 68 0.05 0.00 0.05 69 0.08 0.00 0.08 70 0.07 0.09 0.12 71 0.05 0.06 0.08 72 0.04 0.07 0.08 73 0.09 0.05 0.10 74 0.05 0.03 0.06 75 0.04 0.03 0.05 76 0.07 0.00 0.07 77 0.07 0.01 0.07 78 0.06 0.07 0.09 79 0.07 0.08 0.11 80 0.07 0.06 0.09 81 0.07 0.08 0.11 6
KIT DATASET INTRA-ASSAY SD INTER-ASSAY SD TOTAL ASSAY SD AR 82 0.06 0.06 0.08 83 0.04 0.05 0.07 84 0.12 0.06 0.13 85 0.09 0.06 0.11 86 0.09 0.09 0.13 87 0.04 0.07 0.08 88 0.03 0.02 0.04 89 0.03 0.04 0.05 90 0.06 0.15 0.16 91 0.10 0.09 0.13 92 0.07 0.03 0.08 93 0.05 0.22 0.23 94 0.05 0.05 0.07 RC: ROCHE COBAS RT: ROCHE TAQMAN ASSAY; IH: IN-HOUSE ASSAY; GX: GENEXPERT ASSAY; ME: EASYQ STANDARD PREP; AR: ABBOTT REALTIME ASSAY TABLE 6: MEDIAN LOG10 RECOVERY FOR EACH KIT OVER PAST FOUR ROUNDS, RNA035 (SAMPLES 6-25). Excludes samples <100 copies/ml for RT, ME, GX and AR. Assay Number of Labs* Number of Samples Tested* Median Log Recovery ALL 66 1039 0.0529 AR 39 486 0.0129 AR-RT 61 867 0.0386 RT 32 381 0.0959 * The ALL median log 10 recovery was generated from the data from all laboratories that participated in the four most recent 5-member RNA panels with a commercially available PCR kit; AR: ABBOTT REALTIME; the AR-RT median log 10 recovery was generated from the data from all laboratories that participated in the four most recent 5-member RNA panels with a commercially available real-time PCR kit (specifically, the Abbott RealTime and Roche TaqMan assays); RT: ROCHE TAQMAN. ** Number of labs/samples included in the calculation of the median log recovery value. TABLE 7: THE STANDARD ERROR OF LOG10 RECOVERY OVER PAST FOUR ROUNDS, RNA035 (SAMPLES 6-25), USED IN THE ACCURACY STATISTIC. Excludes samples <100 copies/ml for RT, ME, GX and AR. OBSERVED STANDARD ERROR THRESHOLD STANDARD ERROR STANDARD ERROR USED IN ANALYSIS 0.051940 0.080517 0.080517 7
TABLE 8. ACCURACY SCORING FOR DATASETS GENERATED WITH AN ASSAY THAT DOES NOT HAVE AN ASSAY-SPECIFIC MEDIAN. MEDIAN LOG10 RECOVERY COMPARED TO NOMINAL, A REALTIME MEDIAN AND AN ALL-ASSAY MEDIAN. KIT DATASET Accuracy (Nominal) P-Value Accuracy (RealTime Median) P-Value Accuracy (All-Assay Median) RC 1 2.488 2.009 1.832 2 1.135 0.655 0.478 3 2.310 1.831 1.654 4 0.892 0.413 0.236 5 1.921 1.442 1.265 6 1.176 0.697 0.520 7 0.775 0.295 0.118 8 1.505 1.026 0.848 IH 40 2.016 1.536 1.359 ME 41-0.891-1.370-1.547 42-0.030-0.510-0.687 GX 43 2.510 2.030 1.853 44 1.566 1.086 0.909 45 2.150 1.670 1.493 46 1.637 1.157 0.980 47 1.006 0.526 0.349 48 1.199 0.720 0.543 49-0.503-0.982-1.159 50 0.939 0.460 0.282 P-Value RC: ROCHE COBAS ASSAY; GX: GENEXPERT ASSAY; IH: IN-HOUSE ASSAY; ME: EASYQ STANDARD PREP 8
TABLE 9. PERFORMANCE STATISTICS FOR RNA PROFICIENCY PANELS RNA035 (SAMPLES 6-25) FOR ASSESSMENTS THAT INCLUDE ACCURACY AND THE INTRA-ASSAY STANDARD DEVIATION. (Entries of zero are excluded for clarity) DS: DATASET; VER: ROCHE TAQMAN ASSAY USED (48 OR 96; AmpliPrep or High Pure [HP]) AND VERSION [v2]; INHOUSE ASSAY USED [IH];ABBOTT RT ASSAY USED (0.6ML PROTOCOL [06], AUTOMATED [AUTO] OR MANUAL [MAN] EXTRACTION); MOLECULAR BEACONS EASYQ ASSAY VERSION 2.0 [ME]. NI: THE NUMBER OF INVALID SAMPLES ON THE PANEL; FP: NUMBER OF NEG SAMPLES ABOVE DETECTION LIMIT; FN: NUMBER OF POS SAMPLES BELOW DETECTION LIMIT; NDL: NUMBER OF SAMPLES AT THE LIMIT OF DETECTION; FN50/FN200: NUMBER OF POS SAMPLES AT 50 COPIES/ML BELOW DETECTION LIMIT ON THE ABBOTT RT ASSAY AND THE ROCHE TAQMAN ASSAY/ NUMBER OF POS SAMPLES AT 200 COPIES/ML BELOW DETECTION LIMIT ON THE ME ASSAY; N: THE NUMBER OF SAMPLES INCLUDED IN THE STANDARD DEVIATION; EST: THE TOTAL-ASSAY STANDARD DEVIATION; DIFF: AVERAGE DIFFERENCE BETWEEN THE LOG10 RECOVERY IN A LABORATORY WITH THE EXPECTED LOG10 RECOVERY EXPONENTIATED BACK INTO THE RAW SCALE; ACC: ACCURACY STATISTIC; P: THE P-VALUE FOR EVALUATING FN50/FN200, SD, OR ACCURACY; UNOFF SC: UNOFFICIAL SCORE INCLUDING SCORING FOR SENSITIVITY FOR ABBOTT KITS AND SCORING FOR ACCURACY FOR ROCHE COBAS KITS OFF SC: OFFICIAL SCORE BASED ON PERFORMANCE INCLUDING NON-TECHNICAL SCORING CRITERIA (QUERY RESPONSIVENESS AND INCLUSION OF PHI/PII). THIS EXCLUDES SCORING FOR SENSITIVITY FOR ABBOTT KITS KIT DATASET VER NI NIC FP FN NDL FN50, FN200 TOTAL ASSAY SD ACCURACY UNOFF OFF ACC P SC SC N P N EST P AVE RC 1 ROCH4800 4 12 0.09 1.272 DEV 1.832 C C 2 ROCH6800 4 12 0.07 0.990 0.478 C C 3 ROCH6800 7 21 0.10 1.231 1.654 C C 4 ROCH6800 4 12 0.14 0.946 0.236 C C 5 ROCH6800 4 12 0.08 1.145 1.265 C C 6 ROCH8800 4 12 0.10 0.997 0.520 C C 7 ROCH8800 4 12 0.13 0.926 0.118 C C 8 ROCH8800 4 12 0.11 1.060 0.848 C C RT 9 RT48v2 4 12 0.12 1.050 0.264 C 10 RT48v2 4 12 0.17 0.884 -.666 C 11 RT48v2 4 12 0.11 1.049 0.257 C 12 RT48v2 4 12 0.13 1.212 1.035 C 13 RT48v2 4 12 0.10 1.221 1.075 C 14 RT48v2HP 4 12 0.18 0.440-4.42 0.01 P 15 RT48v2 4 12 0.18 0.970 -.166 C 16 RT48v2HP 4 12 0.17 0.496-3.78 0.05 PC 17 RT48v2 4 12 0.17 0.799-1.21 C 18 RT48v2 4 12 0.14 0.940 -.333 C 19 RT48v2HP 4 12 0.12 0.568-3.05 0.05 PC 20 RT96v2 4 12 0.16 1.086 0.444 C 21 RT96v2 4 12 0.16 1.004 0.020 C 22 RT96v2 4 12 0.11 1.189 0.933 C 9
KIT DATASET VER NI NIC FP FN NDL FN50, FN200 TOTAL ASSAY SD ACCURACY UNOFF OFF ACC P SC SC N P N EST P AVE RT 23 RT96v2 5 3 9 0.13 1.139 DEV 0.701 P 24 RT96v2 4 12 0.17 1.098 0.503 C 25 RT96v2 4 12 0.14 0.976 -.128 C 26 RT96v2 4 12 0.09 1.020 0.108 C 27 RT96v2 4 12 0.13 1.095 0.487 C 28 RT96v2 4 12 0.15 0.951 -.272 C 29 RT96v2 4 12 0.16 1.141 0.711 C 30 RT96v2 4 12 0.18 0.889 -.633 C 31 RT96v2 4 12 0.15 0.876 -.711 C 32 RT96v2 4 12 0.16 0.988 -.067 C 33 RT96v2 4 12 0.20 0.926 -.413 C 34 RT96v2 4 12 0.15 0.820-1.07 C 35 RT96v2 4 12 0.16 1.038 0.203 C 36 RT96v2 4 12 0.13 1.029 0.153 C 38 RT96v2 4 12 0.18 0.970 -.167 C 39 RT96v2 4 12 0.08 1.163 0.816 C IH 40 InHouse 4 12 0.08 1.165 1.359 C ME 41 STDv1 4 12 0.12 0.823 -.891 C 42 STDv1 4 12 0.07 0.965 -.030 C GX 43 GeneXper 4 12 0.08 1.277 1.853 C 44 GeneXper 4 12 0.09 1.072 0.909 C 45 GeneXper 4 12 0.09 1.195 1.493 C 46 GeneXper 4 12 0.10 1.086 0.980 C 47 GeneXper 4 12 0.18 0.966 0.349 C 48 GeneXper 4 12 0.08 1.002 0.543 C 49 GeneXper 4 12 0.19 0.730 -.503 C 50 GeneXper 6 18 0.05 0.954 0.282 C AR 51 06AUTO 4 12 0.06 0.754-1.52 C 52 06AUTO 1 4 1 12 0.14 0.696-1.95 C 53 06AUTO 4 12 0.12 1.143 0.722 C 54 06AUTO 4 12 0.12 1.022 0.117 C 55 06AUTO 4 12 0.07 0.986 -.078 C 56 06AUTO 4 12 0.09 1.005 0.024 C 57 06MAN 4 12 0.20 0.836 -.968 C 58 06AUTO 4 12 0.07 1.216 1.055 C 59 06AUTO 4 12 0.17 1.163 0.815 C 60 06AUTO 4 12 0.11 0.975 -.137 C 61 06AUTO 1 4 12 0.11 0.983 -.093 PC 10
KIT DATASET VER NI NIC FP FN NDL FN50, FN200 TOTAL ASSAY SD ACCURACY UNOFF OFF ACC P SC SC N P N EST P AVE AR 62 06AUTO 4 12 0.09 1.064 DEV 0.336 C AR 63 06MAN 4 12 0.11 1.018 0.095 C AR 64 06AUTO 4 12 0.05 0.911 -.503 C AR 65 06AUTO 4 12 0.05 1.075 0.391 C AR 66 06AUTO 4 12 0.08 0.899 -.571 C AR 67 06MAN 4 12 0.12 1.364 1.673 C AR 68 06MAN 4 12 0.05 1.001 0.007 C AR 69 06AUTO 4 12 0.08 0.972 -.154 C AR 70 06AUTO 4 12 0.12 0.905 -.538 C AR 71 06MAN 4 12 0.08 0.967 -.183 C AR 72 06AUTO 4 12 0.08 0.958 -.229 C AR 73 06AUTO 4 12 0.10 0.995 -.029 C AR 74 06AUTO 4 12 0.06 0.902 -.559 C AR 75 06AUTO 4 12 0.05 1.076 0.393 C AR 76 06AUTO 4 12 0.07 1.426 1.915 C AR 77 06AUTO 4 12 0.07 1.107 0.549 C AR 78 06AUTO 4 12 0.09 1.166 0.826 C AR 79 06AUTO 4 12 0.11 0.957 -.236 C AR 80 06AUTO 4 12 0.09 1.177 0.880 C AR 81 06MAN 4 12 0.11 1.339 1.576 C AR 82 06AUTO 4 12 0.08 0.935 -.364 C AR 83 06AUTO 4 12 0.07 0.912 -.496 C AR 84 06AUTO 4 12 0.13 0.962 -.206 C AR 85 06MAN 4 12 0.11 0.913 -.490 C AR 86 06MAN 4 12 0.13 1.192 0.948 C AR 87 06AUTO 4 12 0.08 0.935 -.364 C AR 88 06MAN 4 12 0.04 1.023 0.125 C AR 89 06AUTO 4 12 0.05 0.837 -.960 C AR 90 06AUTO 4 12 0.16 0.971 -.161 C AR 91 06AUTO 4 12 0.13 0.925 -.423 C AR 92 06AUTO 1 4 1 12 0.08 1.032 0.172 C AR 93 06AUTO 2 4 2 0.05 12 0.23 0.05 1.272 1.297 P AR 94 06MAN 4 12 0.07 0.916 -.476 C RC: ROCHE COBAS ASSAY; RT: ROCHE TAQMAN ASSAY; IH: IN-HOUSE ASSAY; ME: EASYQ STANDARD PREP; AR: ABBOTT ASSAY DATA SETS WITH PROBLEMS ARE HIGHLIGHTED WITH SHADING. 11
TABLE 10. COMPARISON OF KIT LOT NUMBER, MEAN LOG10 RECOVERY, TOTAL ASSAY SD, AND SCORE FOR DATA GENERATED WITH THE ROCHE TAQMAN ASSAY IN RNA035 (SAMPLES 6-25). SHADED ROWS REPRESENT DATASETS WITH TECHNICAL PROBLEMS IN THIS LATEST ROUND OF TESTING. RED FONT INDICATES LABORATORIES THAT ARE BEING MONITORED BUT DID NOT SUBMIT DATA AND GET SCORED FOR THIS ROUND. VER DATASET KIT LOT MEAN LOG 10 TOTAL ASSAY SD SCORE 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 RT48v2 9 Y15616 Y24080 E00164 E00164 0.15 0.06 0.17 0.09 0.11 0.09 0.10 0.12 C C C C RT48v2 10 Y24080 E00164 E00164 E00164 0.05-0.06 0.16 0.02 0.16 0.16 0.17 0.17 C C C C RT48v2 - Y15616 E00164 EXE OH 0.05-0.03 EXE OH 0.16 0.15 EXE OH C C EXE OH RT48v2 - Y16242 EXE EXE EXE -0.18 EXE EXE EXE 0.23 EXE EXE EXE PC EXE EXE EXE RT48v2 11 Y15616 Y24080 E00164 E00164 0.19 0.01 0.17 0.10 0.11 0.13 0.12 0.11 C C C C RT48v2 12 Y11397 Y18458 Y14818 Y22924 0.21-0.07 0.21 0.13 0.18 0.22 0.11 0.13 C PC C C RT48v2 - Y24080 Y24080 OH OH 0.10 0.09 OH OH 0.14 0.13 OH OH C C OH OH RT48v2 13 Y14818 Y14818 Y21188 Y14818 0.18 0.17 0.21 0.17 0.12 0.11 0.10 0.10 C C C C RT48v2HP 14 - - - - -0.49-0.07-0.28-0.20 0.16 0.19 0.19 0.18 P P P P RT48v2 15 Y15616 Y24080 Y24080 Y14818-0.05 0.03 0.27 0.09 0.12 0.13 0.17 0.18 C C C C RT96v2 - Y10651 OH OH OH 0.08 OH OH OH 0.13 OH OH OH C OH OH OH RT48v2 - X09013 X09013 Y17461 EXT 0.11 0.09 0.21 EXT 0.13 0.13 0.13 EXT C C C EXT RT48v2HP 16 EXE - - - EXE -0.34-0.34-0.09 EXE 0.24 0.22 0.17 EXE P P PC RT48v2 17 Y15616 Y15616 E00164 E00164-0.01-0.10 0.15-0.05 0.16 0.18 0.17 0.17 C C C C RT48v2 18 Y24080 Y24080 E00164 E00164 0.12 0.02 0.09 0.05 0.10 0.15 0.14 0.14 C C C C RT48v2HP 19 - - - -0.25-0.19-0.08-0.08 0.10 0.09 0.11 0.12 PC PC PC PC RT48v2 20 Y15616 Y24080 Y24080 E00164 0.10 0.00 0.21 0.21 0.13 0.14 0.14 0.16 C C C C RT96v2 - EXE EXE OH OH EXE EXE OH OH EXE EXE OH OH EXE EXE OH OH RT96v2 21 Y14632 Y18458 Y21188 Y21188 0.06 0.04 0.21 0.09 0.15 0.18 0.18 0.16 C C C C RT96v2 22 X14181 X14181 Y16242 Y16242 0.15 0.18 0.21 0.15 0.20 0.16 0.13 0.11 C C C C 12
VER DATASET KIT LOT MEAN LOG 10 TOTAL ASSAY SD SCORE 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 RT96v2 23 Y16242 Y10651 Y10651 Y16242 0.13 0.07 0.25-0.16 0.14 0.13 - C C C P RT96v2 24 Y11397 Y17461 Y21188 Y28679 0.23 0.03 0.24 0.05 0.18 0.17 0.15 0.17 C C C C RT96v2 25 X13007 Y18458 Y18458 Y25278 0.18 0.06 0.07 0.02 0.14 0.15 0.13 0.14 C C C C RT96v2 26 Y16242 Y10651 Y10651 Y10651 0.18 0.09 0.14 0.06 0.11 0.09 0.09 0.09 C C C C RT96v2 27 Y11397 Y11397 Y11397 Y17121 0.16 0.03-1.29 0.13 0.17 0.16 1.03 0.13 C C P C RT96v2 28 Y14632 Y28460 Y23998 E00588 0.16-0.01 0.16-0.01 0.18 0.15 0.12 0.15 C C C C RT96v2 29 Y14818 Y14818 Y21188 E00588 0.26 0.15 0.26-0.05 0.14 0.12 0.12 0.16 C C C C RT96v2 - Y03434 Y03434 Y17121 EXT 0.28 0.26 0.29 EXT 0.15 0.16 0.12 EXT C C C EXT RT96v2 30 Y16436 Y16436 Y17461 Y25278-0.12 0.02 0.26 0.11 0.19 0.18 0.21 0.18 C C PC C RT96v2 - X09013 X09013 Y2399800 EXT 0.06 0.09 0.12 EXT 0.12 0.12 0.12 EXT C C C EXT RT96v2 31 Y17461 Y18458 Y22924 E00588 0.11 0.00 0.02 0.02 0.17 0.17 0.15 0.15 C C C C RT96v2 32 Y17121 Y20212 E00588 Y25278 0.16 0.05 0.17-0.01 0.16 0.15 0.15 0.16 C C C C RT48v2 33 Y10651 Y16436 Y16436 Y18458 0.10-0.06 0.22 0.00 0.17 0.19 0.19 0.20 C C C C RT96v2 34 X05514 Y17121 Y20212 Y20212 0.11-0.03 0.06-0.10 0.17 0.17 0.17 0.15 C C C C RT96v2 35 Y10651 Y10651 Y08928 Y25278 0.10 0.03 0.26 0.05 0.15 0.14 0.16 0.16 C C C C RT96v2 36 Y06193 Y14632 Y21188 Y22924 0.19-0.25 0.12 0.01 0.18 0.21 0.11 0.13 C PC C C RT96v2 - X1300700 X13007 Y11397 - -0.08 0.08 0.14-0.12 0.15 0.17 - C C C - RT96v2 38 Y16436 Y17461 Y22924 E00588 0.01 0.13 0.22-0.03 0.13 0.15 0.14 0.08 C C C C RT96v2 39 X06831 E00164 E00164 E00164 0.21 0.12 0.17 0.14 0.10 0.08 0.06 0.08 C C C C 13
TABLE 11. COMPARISON OF KIT LOT NUMBER, MEAN LOG10 RECOVERY, TOTAL ASSAY SD, AND SCORE FOR DATA GENERATED WITH THE ABBOTT REALTIME ASSAY IN RNA035 (SAMPLES 6-25). SHADED ROWS REPRESENT DATASETS WITH TECHNICAL PROBLEMS IN THIS LATEST ROUND OF TESTING. RED FONT INDICATES LABORATORIES THAT ARE BEING MONITORED BUT DID NOT SUBMIT DATA AND GET SCORED FOR THIS ROUND. VER DATASET KIT LOT MEAN LOG 10 TOTAL ASSAY SD SCORE 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 AR0.6AUT 51 480594 480594 480538 480538-0.15-0.04-0.13-0.12 0.10 0.10 0.06 0.06 C C C C AR0.6AUT 52 480594 480594 480594 487982-0.10-0.32-0.13-0.03 0.07 0.16 0.13 0.14 C C C C AR0.6AUT 53 480537 482420 485089 482420 0.19 0.01-0.05 0.13 0.11 0.11 0.12 0.12 C C C C AR0.6AUT 54 480291 481998 481998 483548 0.17-0.06-0.01-0.02 0.10 0.11 0.12 0.12 C C C C AR0.6AUT 55 478983 478983 478065 478065 0.07 0.03-0.06-0.02 0.07 0.05 0.07 0.07 C C C C AR0.6AUT 56 480461 480461 480461 480461 0.11 0.06-0.09-0.01 0.06 0.06 0.09 0.09 C C C C AR0.6MAN 57 481685 481685 481685 487988-0.15-0.03-0.18 0.10 0.16 0.18 0.17 0.20 C C C C AR0.6AUT 58 478151 480460 479390 483847 0.12 0.16 0.10 0.01 0.08 0.05 0.05 0.07 C C C C AR0.6AUT 59 480460 480460 481512 481998 0.21 0.08-0.13 0.16 0.12 0.10 0.16 0.17 C C C C AR0.6AUT 60 479045 478983 482609 483361 0.01-0.11 0.12-0.01 0.10 0.12 0.13 0.11 C C C C AR0.6AUT 61 478984 478984 478984 485081 0.08 0.06-0.15 0.03 0.09 0.08 0.12 0.11 PC C PC PC AR0.6AUT 62 483978 485508 487561 487561-0.06-0.05 0.04-0.04 0.12 0.13 0.11 0.05 C C C C AR0.6AUT 63 480460 480460 482418 483974-0.06 0.04 0.12 0.06 0.08 0.08 0.09 0.09 C C C C AR0.6MAN 64 480460 480460 482418 483974-0.09-0.17 0.04-0.02 0.08 0.11 0.07 0.11 C C C C AR0.6AUT 65 469411 478845 478845 478845 0.03 0.08 0.04 0.03 0.03 0.04 0.04 0.05 C C C C AR0.6AUT 66 478664 478665 483864 485081 0.05 0.00-0.09-0.10 0.13 0.13 0.07 0.08 C C C C AR0.6MAN 67 480463 480463 484888 486856 0.28 0.22 0.06 0.03 0.11 0.10 0.12 0.12 C C C C AR0.6MAN 68 481996 481996 485089 485089-0.01 0.00 0.04 0.02 0.06 0.05 0.05 0.05 C C C C AR0.6AUT 69 480066 480461 480945 480945 0.03-0.06 0.00 0.03 0.08 0.07 0.08 0.08 C C C C AR0.6AUT 70 479047 483513 483513 483513 0.01 0.09-0.07-0.15 0.07 0.05 0.08 0.12 PC C C C AR0.6MAN 71 479047 483513 483513 483513 0.00-0.10-0.03-0.03 0.07 0.09 0.06 0.08 PC C C C AR0.6AUT - 480291 480460 479390 OH -0.02-0.05-0.07 OH 0.04 0.04 0.06 OH C C C OH 14
VER DATASET KIT LOT MEAN LOG 10 TOTAL ASSAY SD SCORE 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 AR0.6AUT 72 480291 480460 479390 481998-0.08 0.03 0.08-0.05 0.13 0.13 0.08 0.08 C C C C AR0.6AUT 73 480291 480460 481512 483513 0.10-0.01-0.07 0.01 0.07 0.06 0.08 0.10 C C C C AR0.6AUT 74 479390 480460 481512 482739 0.02-0.05-0.02-0.07 0.07 0.07 0.07 0.06 C C C C AR0.6AUT 75 480291 480460 479390 481512 0.04 0.05 0.09 0.00 0.10 0.11 0.12 0.05 C C C C AR0.6AUT 76 480594 480594 482912 486849 0.14 0.14 0.18 0.21 0.11 0.11 0.05 0.07 C C C C AR0.6AUT 77 481140 481140 478983 484888 0.10 0.06 0.26 0.00 0.07 0.06 0.05 0.07 C C C C AR0.6AUT 78 479383 483864 483864 483864 0.13-0.02 0.15 0.06 0.17 0.14 0.13 0.09 C C C C AR0.6AUT 79 474399 474399 485081 485081-0.05-0.12 0.09 0.06 0.07 0.07 0.10 0.11 C C C C AR0.6AUT - 476172 476172 476172 EXT -0.04 0.02-0.02 EXT 0.09 0.09 0.06 EXT C C C EXT AR0.6AUT 80 479046 EXE EXE 482573 0.18 EXE EXE 0.02 0.10 EXE EXE 0.09 C EXE EXE C AR0.6MAN 81 479046 EXE EXE 482573 0.15 EXE EXE 0.24 0.09 EXE EXE 0.11 C EXE EXE C AR0.6AUT 82 480291 480460 479390 482739 0.06-0.08 0.03-0.07 0.16 0.16 0.17 0.08 C C C C AR0.6AUT 83 478983 479626 482609 483361-0.09-0.05-0.02 0.05 0.06 0.06 0.06 0.07 PC C C C AR0.6AUT 84 478984 480463 478984 480463 0.04 0.02-0.14 0.06 0.09 0.07 0.11 0.13 C C C C AR0.6MAN 85 478984 480463 478984 480463 0.02 0.11-0.13-0.03 0.09 0.06 0.11 0.11 C C C C AR0.6MAN 86 479047 479047 479047 479047 0.12 0.22 0.02-0.01 0.09 0.11 0.11 0.13 C C C C AR0.6AUT 87 EXE 479390 481512 481998 EXE -0.11-0.02 0.02 EXE 0.12 0.09 0.08 EXE C C C AR0.6MAN 88 480461M 480461M 480461M 486849 0.01 0.05 0.05-0.01 0.04 0.05 0.04 0.04 C C C C AR0.6AUT 89 482418 483864 483864 483864-0.13-0.06-0.03-0.04 0.13 0.10 0.08 0.05 C C C C AR0.6AUT 90 480594 480594 486730 486730 0.19-0.18-0.01 0.01 0.12 0.16 0.16 0.16 C C C C AR0.6AUT - 479850 479850 479850 EXT -0.22-0.06-0.07 EXT 0.15 0.12 0.11 EXT C C C EXT AR0.6AUT 91 480066 481996 481996 485795 0.13-0.09-0.10-0.02 0.12 0.13 0.15 0.13 C C C C AR0.6AUT - 479390 480460 481512 EXT -0.02 0.05-0.14 EXT 0.04 0.05 0.09 EXT C C C EXT AR0.6AUT 92 478151 480460 481512 483847 0.10-0.37-0.01 0.02 0.13 0.20 0.10 0.08 C PC C C AR0.6AUT 93-481490 476172 481490 0.11 0.12-0.16 0.40 0.19 0.20 0.18 0.23 C C PC P AR0.6MAN 94 477901 478665 485509 483974 0.00 0.05-0.08-0.07 0.10 0.07 0.07 0.07 C C C C 15
TABLE 12. COMPARISON OF KIT LOT NUMBER, MEAN LOG10 RECOVERY, TOTAL ASSAY SD, AND SCORE FOR DATA GENERATED WITH THE ROCHE COBAS ASSAY IN RNA035 (SAMPLES 6-25). SHADED ROWS REPRESENT DATASETS WITH TECHNICAL PROBLEMS IN THIS LATEST ROUND OF TESTING. RED FONT INDICATES LABORATORIES THAT ARE BEING MONITORED BUT DID NOT SUBMIT DATA AND GET SCORED FOR THIS ROUND. VER DATASET KIT LOT MEAN LOG 10 TOTAL ASSAY SD SCORE 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 ROCH4800 1 - - - - 0.27-0.17 0.20 0.09 0.09 0.09 0.09 C P C C ROCH6800 2 - - - - 0.04 0.06 0.12 0.15 0.09 0.08 0.07 0.07 C C C C ROCH6800 3 - - - - 0.07 0.09 0.10 0.10 C C ROCH6800 4 - - - - 0.10 0.08-0.07 0.19 0.18 0.17 0.12 0.14 C C C C ROCH6800 5 - - - - 0.06 0.23 0.16 0.16 0.09 0.10 0.09 0.08 C C C C ROCH8800 6 - - - - -0.03 0.17 0.17 0.16 0.13 0.10 C C C ROCH8800 7 - - - - 0.12 0.12 0.23 0.11 0.11 0.13 C C C ROCH8800 8 - - - - 0.04 0.23 0.05 0.16 0.13 0.12 0.13 0.11 C C C C 16
TABLE 13. COMPARISON OF KIT LOT NUMBER, MEAN LOG10 RECOVERY, TOTAL ASSAY SD, AND SCORE FOR DATA GENERATED WITH THE GENEXPERT ASSAY IN RNA035 (SAMPLES 6-25). SHADED ROWS REPRESENT DATASETS WITH TECHNICAL PROBLEMS IN THIS LATEST ROUND OF TESTING. RED FONT INDICATES LABORATORIES THAT ARE BEING MONITORED BUT DID NOT SUBMIT DATA AND GET SCORED FOR THIS ROUND. VER DATASET KIT LOT MEAN LOG 10 TOTAL ASSAY SD SCORE 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 35-2 35-3 35-4 35-5 GeneXpert 43 - - - - 0.22 0.16 0.17 0.17 0.19 0.20 0.10 0.08 C C C C GeneXpert 44 - - - - 0.06 0.13 0.08 0.23 0.04 0.05 0.05 0.09 C C C C GeneXpert 45 - - - - 0.11 0.15 0.24 0.20 0.08 0.04 0.08 0.09 C C C C GeneXpert 46 - - - - 0.14 0.24 0.09 0.06 0.05 0.10 0.09 0.10 C C C C GeneXpert - - - - - 0.17 0.16 0.05 0.05 C C GeneXpert 47 - - - - -0.13 0.16 0.18 0.11 0.17 0.17 0.18 0.18 C C C C GeneXpert 48 - - - - 0.03 0.12 0.17 0.07 0.07 0.07 0.08 0.08 C C C C GeneXpert 49 - - - - 0.08 0.12-0.06-0.30 0.09 0.07 0.10 0.19 C C C C GeneXpert - - - - - 0.04 0.04 C GeneXpert - - - - - 0.21 0.15 0.19 EXT 0.05 0.04 0.04 EXT C C C EXT GeneXpert 50 - - - - 0.05 0.10 0.06 0.05 C C 17
Mean LOG 10 RNA Recovery Mean LOG 10 RNA Recovery VIROLOGY QUALITY ASSURANCE PROGRAM FIGURE 1. LINE PLOT OF MEAN LOG10 RECOVERY FOR DATA GENERATED WITH THE ROCHE TAQMAN ASSAY IN RNA028 SAMPLES, RNA029 SAMPLES, RNA030 SAMPLES, RNA031 SAMPLES, RNA032 SAMPLES, RNA033 SAMPLES AND RNA035 SAMPLES 1-25. LABORATORIES WITHOUT ( left panel ) and WITH (right panel) SCORING PROBLEMS. Taqman: Labs with all C scores Taqman: Labs with at least one PC or P score 0.4 Kits S02574 and higher High Pure Kits Adjusted Means Other Kits 0.4 Kits S02574 and higher Adjusted Means value= 0.467 High Pure Kits Other Kits 0.2 0.2 0.0 0.0-0.2-0.2-0.4-0.4 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-25 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-25 18
Mean LOG 10 RNA Recovery Mean LOG 10 RNA Recovery VIROLOGY QUALITY ASSURANCE PROGRAM FIGURE 2. LINE PLOT OF MEAN LOG10 RECOVERY FOR DATA GENERATED WITH THE ABBOTT ASSAY IN RNA028 SAMPLES, RNA029 SAMPLES, RNA030 SAMPLES, RNA031 SAMPLES, RNA032 SAMPLES, RNA033 SAMPLES AND RNA035 SAMPLES 1-25. LABORATORIES WITHOUT (left panel) and WITH (right panel) SCORING PROBLEMS. 0.4 Abbott: Labs with all C scores Adjusted Means 0.4 Abbott: Labs with at least one PC or P score value= 0.465 Adjusted Means value= 0.416 0.2 0.2 0.0 0.0-0.2-0.2-0.4-0.4 value= -0.896 value= -1.497 value= -0.697 value= -0.925 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-25 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-25 19
Mean LOG 10 RNA Recovery Mean LOG 10 RNA Recovery VIROLOGY QUALITY ASSURANCE PROGRAM FIGURE 3. LINE PLOT OF MEAN LOG10 RECOVERY FOR DATA GENERATED WITH THE GENEXPERT (left panel) and ROCHE COBAS (right panel) ASSAYs IN RNA029 SAMPLES, RNA030 SAMPLES, RNA031 SAMPLES, RNA032 SAMPLES, RNA033 SAMPLES AND RNA035 SAMPLES 1-25. ALL LABORATORIES INCLUDED. GeneXpert: All Labs Roche Cobas: All Labs 0.4 Adjusted Means 0.4 Adjusted Means 0.2 0.2 0.0 0.0-0.2-0.2-0.4-0.4 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-25 RNA 028 RNA 029 RNA 030 RNA 031 RNA 032 RNA 033 RNA 035 1-25 20