Developing a Validated Tool For Evaluating Whole Slide Images

Developing a Validated Tool For Evaluating Whole Slide Images

Authors Russell Silowash, BS Evaluation Team Research Analyst Leslie Anthony, MA UPMC IMITS Telepathology Project Project Manager Robb Wilson, MA Evaluation Team Project Manager Dana Grzybicki, MD, PhD Evaluation Team Principal Investigator

Background Through appropriations in the defense-spending bills for 2002 and 2004, the University of Pittsburgh Medical Center (UPMC) and the United State Air Force Medical Service (AFMS) created a partnership called the Integrated Medical Information Technology System (IMITS) Program Telepathology is a branch of the IMITS program that implements and validates digital pathology practices

Introduction Whole Slide Imaging (WSI) systems are increasingly popular within pathology practices It is essential to acquire images that are diagnostically equivalent to glass slides There are no validated assessments for evaluating WSI cases The UPMC Digital Pathology Imaging Group is working on the validation of a unique evaluation tool

Participants 5 pathologists 2 pathology fellows 3 staff pathologists with training in GU pathology

Apparatus Prostate tissue viewed on Aperio ImageScope browser BX45 used to view glass slides Aperio T2 robotic scanner Aperio ImageScope browser used to view digital images of slides

Case Selection 30 difficult prostate biopsy foci Diagnosis Number of Cases Adenocarcinoma 12 Atypical 6 Atypical and High Grade PIN 1 Atypical PIN 1 High Grade PIN 2 Benign 8

Hypotheses Internal Validity There will be a positive correlation between the number of slides/images in a case and the time needed to complete the case External Validity Whole slide image quality will be positively correlated with glass slide quality

Hypotheses (continued) Construct Validity There will be a negative correlation between the diagnostic confidence of a participant and the case complexity rating Content Validity The assessment is asking the proper questions for the study at hand

Time Results TIME BREAKDOWN BY PHASE NUMBER OF CASES 35 30 25 20 15 10 30 30 26 26 17 9 29 29 28 21 14 7 11 PATH 1 PATH 2 PATH 3 PATH 4 PATH 5 5 0 4 4 4 4 2 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 < 15 MIN 15-30 MIN 30-45 MIN MISSING < 15 MIN 15-30 MIN 30-45 MIN MISSING WSI PHASE TIME VALUES GLASS Most cases took less than 15 minutes to complete for both WSI and glass phases

Internal Validity Results Glass Time Value WSI vs Glass Time Per Case Correlation 1.5 1.4 1.3 y = 0.1686x + 0.8964 1.2 1.1 1 0.9 0.9 1 1.1 1.2 1.3 1.4 1.5 WSI Time Value Positive correlation exists between WSI time per case and glass time per case Statistically significant for only one participant (r 2 =0.327, p<0.01)

External Validity Results Slide Quality Rating 3.5 3 2.5 2 1.5 1 Average Image vs Slide Quality Correlation y = 0.5505x + 0.6673 A positive correlation exists between WSI quality and glass slide quality 0.5 0 0 0.5 1 1.5 2 2.5 3 3.5 Im age Quality Rating

External Validity Results (continued) WSI vs. Glass Quality Correlation Pathologist Spearman s Coefficient 1 0.339* 2-0.016 3 0.400* 4 0.154 5 0.456* *Statistically significant correlations (p<0.01) There were statistically significant positive correlations for 3 of the 5 subjects

Construct Validity Results Confidence In Diagnosis 3.5 3 2.5 2 1.5 WSI Average Case Complexity vs Diagnostic Confidence y = -0.6775x + 3.7803 Negative correlation exists in WSI phase of study between confidence in diagnosis and case complexity 1 0.9 1.4 1.9 2.4 2.9 Case Complexity

Construct Validity Results (continued) Confidence in Diagnosis 3.4 2.9 2.4 1.9 1.4 Glass Average Case Complextiy vs Diagnostic Confidence y = -0.4087x + 3.3959 Negative correlation exists in the Glass phase between case complexity and diagnostic confidence 0.9 0.9 1.4 1.9 2.4 2.9 Case Complexity

Construct Validity Results (continued) Case Complexity vs. Confidence In Diagnosis Pathologist WSI Phase Spearman s Coefficient Glass Phase Spearman s Coefficient 1-0.317 0.018 2-0.729* -0.754* 3-0.562* -0.380* 4-0.913* -0.906* 5-0.187-0.458* *Statistically significant correlations (p<0.01) 3 of 5 pathologists had statistically significant positive correlations in the WSI Phase 4 of 5 pathologists had statistically significant positive correlations in the Glass Phase

Content Validity Results Content validity has been obtained by gaining feedback from pathologists that are part of the Digital Pathology Imaging Group (DPIG)

Summary We were able to establish external and construct validity, however internal validity is rather low. Low internal validity could be due to time categories being too broad Slide quality correlates positively with image quality, which suggests good/bad slides causes good/bad images WSI may be used for more clinical applications

Limitations Only used 5 pathologist participants from UPMC who have had experience with WSI Only prostate tissue examined

Next Steps Change the categories for the time variable Time to Complete (Minus Interruptions) Less Than 15 Minutes 15 30 Minutes 30 45 Minutes 45 60 Minutes Over 60 Minutes Time to Complete (Minus Interruptions) Less Than 30 Seconds 7 10 Minutes 30 Seconds 1 Minute 10 12 Minutes 1 2 Minutes 12 15 Minutes 2 5 Minutes Over 15 Minutes 5 7 Minutes Implement an automatic timing solution

Next Steps (continued) Recruit more pathologists to participate in our validation studies Other educational/medical institutions Air Force Expand across multiple tissue types

Acknowledgements We would like to acknowledge all of the members of the Digital Pathology Special Interest Group for their input and contributions towards this project, especially the Principal Investigator, Drazen Jukic,MD

Funding This work was supported by funding from the U.S. Air Force administered by the U.S. Army Medical Research Acquisition Activity, Fort Detrick, Maryland (Award No. W81XWH-04-2- 0030 and Contract No. DAMD 17-0302-0017). The content of the information does not imply U.S. Air Force or Government endorsement of factual accuracy or opinion