Question: Should smartphone apps be used clinically as oximeters? Answer: No.

April 1, 2020

Professor Lionel Tarassenko¹
Professor Trisha Greenhalgh² 

On behalf of the Oxford COVID-19 Evidence Service Team
¹Department of Engineering Science, University of Oxford
²Nuffield Department of Primary Care Health Sciences
University of Oxford

Correspondence to lionel.tarassenko@eng.ox.ac.uk

VERDICT

There is no evidence that any smartphone technology is accurate for the measurement of blood oxygen saturation for clinical use. Furthermore, the scientific basis of such technologies is questionable. Oxygen saturation levels obtained from such technologies should not be trusted in the clinical assessment of patients.

UPDATE
This review was updated on 19^th May to take account of feedback received from academic colleagues. Whilst the feedback resulted in some minor changes to the text below, it does not change the conclusions. The feedback and our detailed response to it can be viewed here.

BACKGROUND
The COVID crisis is requiring us to manage patients with as little in-person contact as possible. The assessment of a patient with respiratory problems usually includes measurement of blood oxygen saturation (abbreviated SpO₂), using a validated pulse oximeter. This is particularly important in unwell patients with COVID-19, since hypoxia is a serious warning sign for severe pneumonia.¹ Whilst in-person assessment would use a standard pulse oximeter on the patient’s finger, few patients have such a technology in their homes. Various technology companies have developed smartphone apps that are marketed as accurate for measuring oxygen saturation.

A previous rapid review by our team on assessing shortness of breath in remote consultations² turned up two academic papers which claimed to have validated smartphone technologies for measuring oxygen levels in the blood.^{3 4}  Both papers (summarised in a table in the appendix), described comparison against a reference method (finger pulse oximetry or arterial blood gas). Whilst those papers included claims that there was good correlation between the smartphone reading and the reference standard, we were concerned about the risks of relying on these two small studies.² We sought expert input from a Professor of Electrical Engineering (LT) who specialises in medical devices. This paper summarises Professor Tarassenko’s advice.

THE SCIENTIFIC BASIS OF OXYGEN SATURATION MEASUREMENT
Oxygen saturation is the fraction of oxygenated haemoglobin relative to the total haemoglobin (oxygenated + deoxygenated) in the blood.

SpO₂ = [HbO₂] / ([Hb] + [HbO₂])

where HbO₂ is oxygenated haemoglobin and Hb is deoxygenated haemoglobin.

The measurement of SpO₂ with a pulse oximeter relies on the fact that the two forms of the haemoglobin molecule, Hb and HbO₂, have light absorption properties which vary with wavelength in the visible and infra-red parts of the spectrum. It therefore requires the measurement of light transmission or reflection from a body segment such as a finger at two different wavelengths (usually in the red and infra-red).

CRITICAL APPRAISAL OF THE DIGIDOC APP TESTED BY TOMLINSON ET AL³
Despite the principles set out above, the DigiDoc app used in the Tomlinson study claims to measure oxygen saturation levels with just the flash light and camera of a smartphone. The app was reviewed in a blog on the Medpage Today website in 2015 (https://www.medpagetoday.com/blogs/iltifathusain/51888) under the headline “There are apps making unjustifiable claims, exposing patients to unnecessary risk”. The author concludes his review with the following words: “I’d urge DigiDoc to either take the app off the market until the company can support its claims or at least make significant changes to the app store description. In the meantime, clinicians […] should advise their patients not to use it.”

The claims made by DigiDoc are scientifically unsound. The app “measures oxygen saturation within 90-100% with an accuracy of 0-4 RSM compared to a medical grade oximeter”. It is not clear what RSM is (is it root root-mean-square error, RMSE?) but if we assume that they are claiming an error of ±4%, then a random number generator with a mean value of 95% and errors randomly distributed between -4% and +4% would give values between 91% and 99%.

Careful analysis of the paper by Tomlinson et al³ confirms the lack of scientific credibility for the DigiDoc app. Figure 2 in the paper (Bland-Altman plot for the SpO₂ values for the DigiDoc app and the triage pulse oximeter) shows that all the readings, bar two, were between 97% and 100%, i.e. completely normal, with 95% Limits of Agreement at -4% and +3.5%. The authors observe that “reliability is low for the camera-based app, even when the two investigators tested the same patient within 1 to 2 min of each other”. Their statement that the difference between the camera-based app and the triage pulse oximeter is ±4 points should have led them to conclude, given that the x-axis in the Bland-Altman plot only extends from 96% to 100%, that the app was highly inaccurate.

CRITICAL APPRAISAL OF 2019 SCIENTIFIC ARTICLE BY DIGIDOC AND COMPUTER SCIENCE & ENGINEERING DEPARTMENT FROM SMU, DALLAS
On 19^th December 2019, DigiDoc Technologies announced in a blog that a new scientific article (Measuring Oxygen Saturation with Smartphone Cameras using Convolutional Neural Networks) had been published in the November issue of the IEEE Journal of Biomedical & Health Informatics. The three authors of this paper are researcher Xinyi Ding and Associate Professor Eric C. Larson of Southern Methodist University, Dallas and the CEO of DigiDoc Technologies, Dr Damoun Nassehi.

The methodology has a number of limitations, some of which are pointed out in the paper by the authors, but which are laid out more explicitly here:

1. The paper describes the use of a 1D convolutional neural network (CNN) for regressing oxygen saturation. According to the authors, the “bias” (the output of a low-pass filter, which is a measure of the energy of the signal) of the red photoplethysmography (PPG) signal decreases when oxygen saturation decreases. This is true, but the amplitude of the decrease will of course be different when using the app on a finger from a different individual with different skin type and different capillarity.
2. The CNN is trained on 38 subjects and tested on just one. This is repeated 38 times, with a different test subject in each case (leave-one-out methodology). The test dataset when training a neural network should have at least 20% of the subjects, ideally 40% or 50%. Only then can one have confidence that the neural network will give accurate results on new subjects beyond those included in the original study.
3. Pulse oximeters are calibrated to a range of oxygen saturations from 70% to 100%, with blood samples regularly drawn from the volunteers taking part in the calibration study, under medical supervision, used to obtain the reference data. In these calibration studies, the subjects are slowly de-saturated, under medical supervision, by breathing gas mixtures with reduced oxygen content (see Guazzi AR, Villarroel M, Jorge J, Daly J, Frise MC, Robbins PA, Tarassenko L. Non-contact measurement of oxygen saturation with an RGB camera. Biomedical optics express. 2015 Sep 1;6(9):3320-38). The breath holding procedure described in the DigiDoc paper is not a proper de-saturation procedure and produces very few SpO₂ values below 85%. As a consequence, there is virtually no data in the 70%-85% range. From the histogram shown on Fig.9 of the paper, it is clear that the overwhelming majority of the SpO₂ values are between 95% and 100%.

To summarise, the app cannot be used clinically because: (a) the training dataset does not appear to have included the full range of skin types (from Fitzpatrick skin type 1 to 6); (b) the training dataset covered a limited range of oxygen saturation values, mostly in the normal range from 95% to 100%, whereas pulse oximeters used clinically should cover the range from 70% to 100%; (c) there is no independent dataset on which the app has been tested.

In e-mail exchanges with Professor Tarassenko, Dr Larson makes a number of points:

• Smartphone apps should not be used for monitoring COVID-19 because they are not proven accurate at lower saturation levels.
• The statement regarding the change of skin type and capillarity is well founded.
• A five-fold cross validation approach was also used, in which 80% of the subjects are used for training and 20% for testing (at repeated intervals such that each remaining 20% of the subject pool are used for testing – i.e. repeated cross validation).  The results of the five-fold analysis were significantly no different from the leave one out results. These five-fold results were removed from the journal paper in order to meet length requirements for the journal (and because they did not change the paper’s conclusion).
• The algorithm is most accurate above 90%.  One reason for this is because of the breath-holding experimental setup, it is difficult to obtain training data below 85%.
• As acknowledged by Ding et al., the authors of the paper never claim that their algorithm is precise enough for clinical use.

In a further e-mail to Professor Tarassenko, Dr Nassehi states that “the [DigiDoc] Pulse Oximeter app is for use by sports users who are interested in knowing their blood oxygenation level (SpO₂) and Heart Rate. The Pulse Oximeter app is NOT INTENDED FOR MEDICAL USE.”

CRITICAL APPRAISAL OF THE SAMSUNG APP TESTED BY TAYFUR ET AL⁴
The Samsung Galaxy series of phones had a red light emitting diode (LED) built into the phone in addition to the flash light and camera. There were no details released by the company of how its app used the LED to estimate oxygen saturation, but it appears from publicity material on YouTube that it worked via a single-wavelength measurement (albeit with a monochromatic light source, the LED) and therefore that oxygen saturation could not be accurately derived from it.

The methodology used by Tayfur and Afacan in their study is sounder than in the Tomlinson study because their reference device is an arterial blood gas (ABG) analyser. The Bland-Altman plot (Figure 4) in their paper shows that most of the oxygen saturation measurements from their Emergency Department patients were between 95% and 100%. For the few patients whose oxygen saturation measurements were between 85% and 93%, the difference between the smartphone estimate and the ABG device varied between -5.5% and +2.5%. In other words, the readings become less accurate as the patient becomes more hypoxic.

CONCLUSIONS

• It is not physically possible to measure SpO₂ using current smartphone technology.
• The two published studies which assessed smartphone oximeter apps (Digidoc and Samsung) raise serious questions about the diagnostic accuracy.

Disclaimer:  the article has not been peer-reviewed; it should not replace individual clinical judgement and the sources cited should be checked. The views expressed in this commentary represent the views of the authors and not necessarily those of the host institution, the NHS, the NIHR, or the Department of Health and social Care. The views are not a substitute for professional medical advice.

AUTHORS
Professor Tarassenko is Professor of Electrical Engineering at the University of Oxford.

Trish Greenhalgh is a Professor of Primary Care Health Sciences, co-Director of the Interdisciplinary Research In Health Sciences (IRIHS) unit, and joint module coordinator on the Knowledge Into Action (KIA) module of the MSc in Evidence Based Health Care.

SEARCH (for further details see²)

We searched EMBASE and PubMed

Embase: We used the Thesaurus search builder with the terms “dyspnea” OR “hypoxia” OR “oximetry” AND “telemedicine” OR “smartphone” OR “telephone”

Pubmed: We performed the following search string using the relevant MeSH terms:

(“Oximetry”[Mesh] OR “Blood Gas Monitoring, Transcutaneous”[Mesh] OR “Dyspnea”[Mesh] OR “hypoxia”[Mesh]) AND (“Telemedicine”[Mesh] OR “Remote Consultation”[Mesh] OR “Smartphone”[Mesh] OR “Telephone”[Mesh])

APPENDIX: DETAILS OF INCLUDED STUDIES
(acknowledging Koot Kotze and Helene-Mari Van Der Westhuizen²)

REFERENCES

1. Greenhalgh T, Koh GCH, Car J. Covid-19: a remote assessment in primary care. Bmj 2020;368:m1182. doi: 10.1136/bmj.m1182 [published Online First: 2020/03/28]
2. Greenhalgh T, Kotze K, Van Der Westhuizen H-M. Are there any evidence-based ways of assessing dyspnoea (breathlessness) by telephone or video? : Oxford COVID-19 Evidence Service Rapid Review. 30th March 2020. Accessed 1.4.20 at https://www.cebm.net/covid-19/are-there-any-evidence-based-ways-of-assessing-dyspnoea-breathlessness-by-telephone-or-video/.
3. Tomlinson S, Behrmann S, Cranford J, et al. Accuracy of smartphone-based pulse oximetry compared with hospital-grade pulse oximetry in healthy children. Telemedicine and e-Health 2018;24(7):527-35.
4. Tayfur İ, Afacan MA. Reliability of smartphone measurements of vital parameters: A prospective study using a reference method. The American journal of emergency medicine 2019;37(8):1527-30.