Pitfalls of repeat testing, illustrated with the House of Commons

September 9, 2020

Richard Stevens, Annette Pluddemann, Jason Oke

On behalf of the Oxford COVID-19 Evidence Service Team
Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences
University of Oxford

Correspondence to: richard.stevens@phc.ox.ac.uk

Verdict
All diagnostic and monitoring tests in medicine are imperfect, giving false positive and/or false negative test results.  It has been shown in many fields that over-frequent testing allows false positive results to accumulate[1]^,[2],[3].

The speaker of the House of Commons has called for daily testing of MPs, so that the house can return to full sitting[4].  Here, we use the hypothetical case of a full and healthy House of Commons, to illustrate how repeat testing, especially frequent repeat testing, can accumulate spurious diagnoses.

Disclaimer: the exact testing scheme proposed for the House of Commons has not, to our knowledge, been specified.  Our analysis is an illustration of the pitfalls of repeat testing in general and should not be interpreted as an assessment of a real-life testing scheme.

Results
For illustrative purposes, we assume 650 healthy Members of Parliament (MP) attend the House of Commons on Monday to Friday and are tested on each day, with a test that has specificity 96.6% (that is, 3.4% of healthy individuals test ‘false positive’).  In this scenario, of 650 healthy MPs tested on day 1, then 23 (3.4%) would be expected to test (false) positive.  Assuming these 23 now self-isolate, when the remaining 627 MPs return the next day, 21 (3.4% of 627) would be expected to test (false) positive, and so on, until at the end of the second working week 190 MPs would be self-isolating: see Figure 1.

We re-ran the analysis assuming a test with specificity 98.9%: see Figure 2.  Under this assumption, by the end of week 2, an expected 68 MPs would be self-isolating.

The false-positive errors only accumulate in this way if tests are repeated without confirmatory tests.  If we suppose that MPs testing positive are able to obtain a confirmatory test from their GP or a testing centre on the same day, and if we suppose that results from such tests are returned within 24 hours, then by the third day the number self-isolating stabilises at 32 MPs in the first scenario (specificity 96.6%) and at 15 MPs in the second scenario (specificity 98.9%).

Methods and assumptions
We assume that a daily testing programme would use a technology with rapid results such as Lateral Flow Assays, which could give results in under an hour[5].  In a systematic review of serological tests, Bastos et al. estimated the specificity of LFIA for covid-19 to be 96.6%: that is, of 100 healthy subjects, 3.4% test positive for COVID[6].   In a more recent review, Dinner et al. estimated the specificity for point-of-care tests in general to be 98.9%[7]. Since this is an illustrative example only, we hypothesized a population of 650 healthy MPs attending the House of Commons from Monday to Friday each week.  Because the recommended length of self-isolation is two weeks, we run the analysis for 10 working days.

When adding in confirmatory tests, we assumed a high accuracy (specificity 99.8%) and for simplicity we assumed a 24-hour turnaround time for test results[8].

Limitations
This illustrative scenario should not be interpreted literally as an appraisal of any actual testing strategy in the House of Commons.  The full details of such a strategy are not yet known.  The situation in the House of Commons is substantially more complex than we have modelled here.  Our analysis imagines all healthy MPs attending the House on all working days, which does not reflect either the complexity of the Parliamentary timetable or the reality of MPs other duties. We have hypothesized a fully healthy population when in practice it is known that some current MPs have already had covid-19.  The properties of the tests in people with previous disease may be different to those without disease.

We have calculated the “expected number” to test positive each day without modelling random variation and uncertainty: in practice, and numbers could be higher or lower than these expectations.

In our analysis of confirmatory tests, we assumed all MPs access such tests the same day, and that all tests are successful: we did not model the proportion of unsuccessful tests.

Conclusions
Naïve testing strategies allow false positive diagnoses to accumulate.  While we have illustrated the problem with a finite (and hypothetical) population, the principles also apply to other populations.  Any country (or employer) considering regular screening of citizens (or employees) should be aware that repeat testing demands a testing methodology with very high specificity: or, the intelligent use of confirmatory tests.

What this means:

• Testing a healthy population always brings the risk of false positive tests for disease.
• The choice of test should take into account false positive risk as well as false negative risk.
• The choice of test(s) for a screening strategy should take into account multiple testing.
• Combining tests intelligently reduces error, but repeating tests naively accumulates error.

Authors

Richard Stevens is an Associate Professor in Medical Statistics at the Nuffield Department of Primary Care Health Sciences and Course Director of the MSc in Evidence Based Health Care (Medical Statistics).

Annette Plüddemann is Senior Research Fellow at the Centre for Evidence Based Medicine and Course Director of the MSc in Evidence Based Health Care.

Jason Oke is a Senior Statistician at the Nuffield Department of Primary Care Health Sciences and Module Coordinator for Statistical Computing with R and Stata (EBHC Med Stats), and Introduction to Statistics for Health Care Research (EBHC), as part of the Evidence-Based Health Care Programme.

Disclaimer:  the article has not been peer-reviewed; it should not replace individual clinical judgement, and the sources cited should be checked. The views expressed in this commentary represent the views of the authors and not necessarily those of the host institution, the NHS, the NIHR, or the Department of Health and Social Care. The views are not a substitute

[1] Glasziou et al. 2008. https://pubmed.ncbi.nlm.nih.gov/18458278/
[2] Keenan et al. 2009 https://pubmed.ncbi.nlm.nih.gov/19406886/
[3] Stevens et al. 2010.  https://journals.sagepub.com/doi/abs/10.1177/0962280209359886
[4] https://www.bbc.co.uk/news/uk-politics-54044624
[5] https://www.cebm.net/covid-19/what-tests-could-potentially-be-used-for-the-screening-diagnosis-and-monitoring-of-covid-19-and-what-are-their-advantages-and-disadvantages/
[6] Bastos et al. 2020. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7327913/
[7] Dinnes et al. 2020. https://doi.org/10.1002/14651858.CD013705
[8] https://www.gov.uk/government/news/975-of-in-person-coronavirus-covid-19-tests-returned-next-day