MAVARIC – a comparison of automation-assisted and manual cervical screening: a randomised controlled trial

Citation

Kitchener, H.C., Blanks, R., Cubie, H., Desai, M., Dunn, G., Legood, R., Gray, A., Sadique, Z. and Moss, S. (2011), "MAVARIC – a comparison of automation-assisted and manual cervical screening: a randomised controlled trial", Clinical Governance: An International Journal, Vol. 16 No. 3. https://doi.org/10.1108/cgij.2011.24816caa.003

MAVARIC – a comparison of automation-assisted and manual cervical screening: a randomised controlled trial

Article Type: Health technology assessment From: Clinical Governance: An International Journal, Volume 16, Issue 3

H.C. Kitchener, R. Blanks, H. Cubie, M. Desai, G. Dunn, R. Legood, A. Gray, Z. Sadique and S. Moss, on behalf of the MAVARIC Trial Study Group

Objectives

Cervical screening currently relies on manually read slides in which the cytoscreener scans the entire slide looking for abnormal cells. This study evaluated technology that assists reading cytology by automatically detecting abnormal fields of view on a slide and presenting these to a cytoscreener on an automated microscope. This could potentially achieve greater sensitivity and productivity, thus saving lives and achieving a more efficient use of the cytology workforce. This study had the following objectives:

• •

  To determine the sensitivity of automation-assisted reading relative to manual reading.

• •

  To determine any added productivity of automated reading.

• •

  To estimate the comparative cost-effectiveness of automated and manual reading.

• •

  To determine the reliability of ‘no further review’ (NFR) without any reading.

Design

Samples were randomised to a paired arm reported by both automated and manual reading and an arm with manual reading only. All of the cytology was liquid based, and the study incorporated randomisation of both widely used liquid-based cytology systems and their respective automated imaging technology; one of which ranks slides in terms of abnormality and will select around one-fifth as requiring NFR.

Setting

The samples were obtained from women undergoing cervical screening in the NHS programme, principally in general practices, in Greater Manchester, UK.

Samples

Samples from 73,266 women were obtained between March 2006 and February 2009; 72,837 were included in the study. Almost all of the women were aged 25-64 years (69,218). Randomisation resulted in 24,566 (33.7 per cent) slides in the manual arm and 48,271 (66.3 per cent) in the paired arm.

Intervention

In the paired arm, automation-assisted reading of slides was performed in addition to manual reading and management determined by the worse result. Low-grade cytological abnormalities were triaged by a human papillomavirus (HPV) test (Hybrid Capture 2®; Qiagen, Crawley, UK) to select women for colposcopy referral. All women with high-grade abnormalities were referred for colposcopy. If cervical intraepithelial neoplasia grade II (CIN2) or worse (CIN2+) was detected, the woman was treated. Additionally, a detailed economic analysis of the cytology reading was undertaken.

Main outcome measures

The primary outcome was the sensitivity of the final automated result relative to that of the final manual result in the paired arm. Secondary outcome measures included an assessment of productivity and estimates of cost-effectiveness, and an evaluation of the reliability of the NFR facility in the Becton Dickinson (BD) FocalPoint® Guided Screener (GS) Imaging System (BD, Franklin Lakes, NJ, USA).

Results

The proportion of abnormal cytology management results by grade were: borderline, 3.6 per cent; mild dyskaryosis, 2.4 per cent; and moderate and severe dyskaryosis combined, 1.22 per cent. These were very similar to England as a whole. The non-negative cytology amounted to 5.47 per cent in the paired arm and 5.52 per cent in the manual-only arm. Within the paired arm the proportion of discordant pairs on final result was 3.8 per cent (1850/48,271); for 1.3 per cent (625/48,271), the discordance was between inadequate and negative. Discordant pairs occurred in both directions with respect to manual and automated reading. There were 192 additional low-grade/HPV-positive abnormalities detected by manual reading only (manual positive/auto negative) and 47 additional high-grade abnormalities detected by manual reading only in the paired arm. The overall referral rate to colposcopy was 4.7 per cent. The proportion with CIN2+ was 1.6 per cent (398/24,566) and 1.5 per cent (707/48,271) for the manual and paired arms respectively (p=0.10). The primary outcome of the relative sensitivity for CIN2+ of automated reading compared with manual reading in the paired arm was 0.92 [95 per cent confidence interval (CI) 0.85 to 0.95]. The relative specificity was 1.006 (95 per cent CI 1.005 to 1.007).

Productivity in terms of the number of slides read per day by primary screeners was estimated to be 60-80 per cent higher for automated reading than for manual reading. The overall costs per case of CIN2+ detected were almost identical between automated and manual reading (£2892, 95% CI £2720 to £3098; and £2838, 95% CI £2676 to £3030 respectively). The overall costs per case of cervical intraepithelial neoplasia grade III (CIN3) or worse (CIN3+) detected are also very similar between automated and manual reading (£4762, 95% CI £4378 to £5245; and £4775, 95% CI £4400 to £5244 respectively). Manual screening is therefore slightly more expensive and effective, and could be considered cost-effective compared with automated reading if decision-makers were willing to pay at least £5,000 each additional case of CIN2+ detected. NFR in the BD FocalPoint GS Imaging System was reported in 22 per cent of slides and was a very reliable indicator of the absence of underlying disease, with only 3.1 per cent of detected CIN2+ being missed by NFR, and even more so if NFR was restricted to routine screening slides. When both savings in staff time to read slides and the additional equipment costs were taken into account, utilising the NFR option generated cost savings. Based on all slides included in the MAVARIC (Manual Assessment Versus Automated Reading In Cytology) study, assessment of the incremental cost per case detected revealed that decision-makers would need to be willing to pay £2,500 per additional case of CIN2+ detected for it to be more cost-effective to read slides manually instead.

Results of the lifetime modelling indicated that when life-years were used as an outcome measure, manual reading was within the £20,000-30,000 per life-year saved range in which the National Institute for Health and Clinical Excellence would neither accept nor reject this technology on cost-effectiveness grounds alone. Modelled results were also estimated for quality-adjusted life-years gained, but these are highly uncertain given the absence of trial evidence on utility values.

Conclusions

The principal finding was that automation-assisted reading was 8 per cent less sensitive than manual in the detection of CIN2+ and 5 per cent less sensitive for CIN3+. To a large extent, this was due to automation-assisted reading failing to detect cases of low-grade abnormalities that were detected in manual reading. The majority of missed cases were due to failure to detect abnormalities presented rather than location-guided errors. Despite the undoubted productivity gains that could be achieved in terms of slide throughput, there do not appear to be sufficient grounds to recommend automation. The slight gain in specificity is not of clinical importance; the positive predictive value (CIN2+) of additional manually read abnormal cytology leading to colposcopy referral would be in line with that of HPV-positive/mild abnormalities currently triaged to colposcopy. Second, given the pricing obtained from the companies and used in this study, the cost-effectiveness of automation-assisted reading is marginal at best, compared with manual reading. Thirdly, there was a general view among the cytoscreeners that they find the automation-assisted reading more monotonous and prefer manual reading.

Although automation-assisted reading did not compare favourably with manual reading, the robust evaluation of the NFR mode of the BD FocalPoint GS Imaging System showed it to be very reliable and able to achieve cost savings in staff time, even if some methods of manual rapid review were maintained for quality control purposes. A significant reduction in the number of slides needing full screening would enhance efficiency and turnaround times.

Were, however, conclusive evidence to emerge in the future that the sensitivity concerns had been resolved and the cost-effectiveness of automation significantly improved, then the recommendation against automation would warrant reconsideration.

Trial registration

This trial is registered as ISRCTN66377374.

Funding

The National Institute for Health Research Health Technology Assessment programme.

© Crown Copyright

H.C. Kitchener is based at the School of Cancer and Enabling Sciences, University of Manchester, St Mary’s Hospital, Manchester, UK

R. Blanks is based at the Cancer Screening Evaluation Unit, The Institute of Cancer Research, Sutton, UK

H. Cubie is based at the Specialist Virology Centre, Royal Infirmary of Edinburgh, Edinburgh, UK

M. Desai is based at the Manchester Cytology Centre, Central Manchester University Hospitals NHS Foundation Trust, Manchester, UK

G. Dunn is based at the Health Sciences Research Group, School of Community Based Medicine, University of Manchester, Manchester, UK

R. Legood is based at the Health Services Research Unit, London School of Hygiene and Tropical Medicine, London, UK and also at the Health Economics Research Centre, University of Oxford, Oxford, UK

A. Gray is based at the Health Economics Research Centre, University of Oxford, Oxford, UK

Z. Sadique is based at Health Services Research Unit, London School of Hygiene and Tropical Medicine, London, UK

S. Moss is based at Cancer Screening Evaluation Unit, The Institute of Cancer Research, Sutton, UK