Device vs. Deliverable: the Pursuit of Digital Pathology Device Validation

When we consider whole slide imaging (“WSI”) devices, we expect high performance in both speed and image quality, together with consistency over time and among devices from the same manufacturer.  Quality control measures employed during manufacturing certainly contribute to performance and consistency, but what is being done to ensure accuracy of the output?  After all, it’s the image that matters for diagnosis and records, right?

The FDA will soon be issuing guidelines for WSI device technical assessments.  These assessments will be required for any device that will be promoted for use in digital pathology, and may be considered a benchmark for traditional imaging systems.  In fact, the guidelines may be adopted more widely for any imaging instrumentation used for research, including slide scanners.

Upon examination of the assessment categories in the draft FDA guidance (see the document HERE), the first tier of assessments would be performed at the individual component level: slide feeder, light source, imaging optics, mechanical scanner movement, digital imaging sensor, image processing software, image composition, image file formats, image review manipulation software, computer environment, and display.  The concept here is that each individual component would be characterized, qualified and validated for performance to some specification, and the entire device – as a “sum of the parts” – would, by extension, carry the validation to its performance.  Again, first tier validation does not address the quality of the output.

The FDA is also recommending a second tier of assessment, qualifying the performance of the device as a whole.  Categorized as a system-level assessment, the FDA assessment guidelines require testing and validation of color reproducibility, spatial resolution, focusing test, whole slide tissue coverage and stitching error.  The good news is that a second tier validation addresses output quality.  The challenge is finding standards for these guidelines.  Targets exist for spatial resolution and focus testing, and many image analysis applications can determine the presence of tissue in an image, even anomalies resulting from poor stitching.

Datacolor CHROMACAL color calibration slide includes color standards that span the visible spectrum.

Figure 1. Datacolor CHROMACAL color calibration slide includes color standards that span the visible spectrum.

The biggest hurdle to date is a color standard (for tier one validation) and a reference slide (for tier two validation). A quick survey of the internet identifies only one color calibration slide (Figure 1; Datacolor CHROMACAL; http://scientific.datacolor.com) that could be adapted for calibrating the digital imaging acquisition device (i.e. camera) as an individual component, and could also be used to calibrate the imaging system as a whole. Ideally a color reference slide should be different from the calibration slide, but recent attempts within the industry have not produced stable, reliable reference devices. In the meantime, the CHROMACAL slide is a readily available and robust tool for ensuring color quality and reproducibility.

Additional areas for assessment are also recommended by the FDA including the user interface, labeling and quality control.  Sometimes referred to as a third tier assessment, quality control includes a stated frequency, specifies the resources required, and the degree of re-assessment to ensure performance of the device.  Since the image is the major deliverable from the imaging systems, digital image accuracy should be a primary goal of device quality control.  With the proper tools (calibration and reference slides), and the software applications to conduct the qualification or re-calibration of a digital pathology device, individual laboratories could readily manage tier 3 validation locally.

So what’s next?  The FDA expects to release the official guidance at the end of the year, and with the digital pathology market expected to reach nearly $6 billion by 2020, manufacturers of these digital pathology devices are considering their options for following and complying with the guideline, so that they don’t lose out on their piece of the pie.

 

Please share your own thoughts below.  Leave a Comment.

 
Comments
Sathya Srinivasan

Thank you for posting the fantastic article about WSI and the factors that has to be considered for FDA approval. Is ChromaCal thinking of developing specific test slides for WSI devices for testing. The test slide for WSI microscopes does not have to be the circles of colors as it is now, but can be stripes or checker patterns of different colors to cover most of the questions raised in the document. There are Resolution Test Targets from Thor labs (http://www.thorlabs.us/newgrouppage9.cfm?objectgroup_id=4338) for example. Please send me a mail at sathya_sr70@hotmail.com and we can discuss about it and using one of the slide scanners that we have to perform some test studies if you are interested. Thank you.
Sathya Srinivasan