Managing Pandemic Responses with Health Informatics – Challenges for Assessing Digital Health Technologies

Summary

Objectives: To highlight the role of technology assessment in the management of the COVID-19 pandemic.

Method: An overview of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems.

Results: Evaluation of digital health technologies for COVID-19 should be based on their technical maturity as well as the scale of implementation. For mature technologies like telehealth whose efficacy has been previously demonstrated, pragmatic, rapid evaluation using the complex systems paradigm which accounts for multiple sociotechnical factors, might be more suitable to examine their effectiveness and emerging safety concerns in new settings. New technologies, particularly those intended for use on a large scale such as digital contract tracing, will require assessment of their usability as well as performance prior to deployment, after which evaluation should shift to using a complex systems paradigm to examine the value of information provided. The success of a digital health technology is dependent on the value of information it provides relative to the sociotechnical context of the setting where it is implemented.

Conclusion: Commitment to evaluation using the evidence-based medicine and complex systems paradigms will be critical to ensuring safe and effective use of digital health technologies for COVID-19 and future pandemics. There is an inherent tension between evaluation and the imperative to urgently deploy solutions that needs to be negotiated.

Keywords

Health information technology - COVID-19 - evaluation studies - program evaluation

1 Digital Health Technology is Essential for Effective Response to the Pandemic

COVID-19 has placed incredible pressure on health systems globally, to provide immediate responses to the outbreak via contact tracing, testing, and surveillance as well as to mitigate its effects on the delivery of other health services. Public health measures including face coverings, personal hygiene and social distancing have been critical to controlling the spread. These measures have shielded hospital services that are managing high severity cases as inpatients with less severe cases being managed via telehealth. The long-term effects of the virus are creating an ongoing burden for health systems [[1]]. Nursing homes and rehabilitation centers are overloaded with patients that need to recover [[2]]. Prevention and treatment of non-communicable diseases such as cancer, cardiovascular diseases and diabetes has been severely disrupted.

Digital health, which is the application of information and communication technology to support health services, can play a central role in the immediate response to the outbreak as well as mitigating its impact. A wide variety of digital health technologies have been adopted and integrated to assist with pandemic planning, surveillance, testing, contact tracing, quarantine, and health care [[3], [4]]. They feature prominently amongst measures that are being deployed globally, particularly in countries that have maintained low COVID-19 per-capita mortality rates such as China, Taiwan, Singapore, South Korea, Vietnam, Iceland and Australia [[4]–[6]]. The role of digital health was formally highlighted by the Riyadh Declaration which calls on the global health community to “create the infrastructure needed to share effective digital health evidence-based practices and high-quality, real-time data locally and globally to provide actionable information to more health systems and countries” [[7]]. The declaration usefully identifies seven priorities for digital health including health intelligence, scalability and interoperability, artificial intelligence, communication, data governance, quality and effectiveness of digital technology, and research and innovation.

While the Riyadh Declaration emphasises a need for data-driven and evidence-based approaches, it is silent about the role of technology assessment which seeks to evaluate the effects of digital health solutions to ensure safe and effective deployment. Take, for instance, digital contact tracing where global positioning systems, real-time monitoring of mobile devices and wearables are used to identify and track individuals who come into contact with an infected person. While use of digital technologies can speed up the identification of exposed individuals for testing and quarantine, the risks of digital contact tracing including safety and privacy concerns need to be monitored and managed. For instance, digital contact tracing solutions could breach privacy; may result in false alerts or fail to detect individuals if mobile devices are turned off or loose connectivity. Indeed, many digital health solutions for COVID-19 are today being rapidly deployed in health services and to consumers without rigorous evaluation of their safety and effectiveness [[3],[8]]. Evaluation of these deployments can help to identify challenges and mitigate risks before they may have adverse consequences for patient safety and organisational functioning.

The objective of the current paper is to draw attention to the role of technology assessment in the management of the pandemic. We argue that it is necessary to: i) take an evidence-based approach to digital health solutions to ensure they are safe and effective [[9]]; ii) evaluate technology prior to deployment at the front line; and iii) ensure evaluation is rapid and pragmatic, including iterative improvement cycles to ensure solutions have a plausible chance of being successfully deployed within the fast-changing environment posed by the pandemic. An overview of existing research and evaluation approaches was combined with expert perspectives from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics, as well as the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. We argue for the need to combine the evidence-based medicine and complex systems paradigms in technology assessment, and then show how this approach can be applied to evaluate digital health technologies for COVID-19.

2 Approaches to Evaluating Digital Health Technology

Evaluation is needed at each distinct stage in the IT lifecycle including design, development, selection, implementation, use, and ongoing surveillance [[10]]. As we progress through the IT lifecycle stages the focus of evaluation changes from formative assessment ’which is focussed on shaping the design of an intervention such as digital contact tracing and tends to be lightweight, inexpensive and quick’to summative assessment which is concerned with rigorously determining whether the intervention, once in use, actually makes a difference to measures outcomes in the way expected. Therefore methods must be chosen accordingly. For instance, rapid usability testing can be used to refine the design of a digital contact tracing app, while the effectiveness of digital contact tracing over manual methods will require a rigorous approach to measure speed and accuracy.

One possible approach to evaluation is to combine evidence-based medicine and complex systems paradigms as for the wider public health response to COVID-19 [[11]]. In this approach, the evidence-based medicine paradigm which rests on a linear model of causality, is seen to be more suitable for focused research questions around drugs and vaccines that can be framed as ‘PICO’’population-intervention-comparison-outcome’ and evaluated through controlled trials. For broader knowledge gaps, particularly population-wide public health interventions such as handwashing, social distancing and face coverings the complex systems paradigm is recommended. Here the model of causality is emergent, allowing evaluators to account for multiple factors including technological, social, the organizational and wider macroenvironment, interacting in dynamic and unpredictable ways [[12]]. Naturalistic methods, rapid-cycle evaluation and pragmatic trials are described as being more appropriate for supporting widespread and sustained behaviour change across an entire population.

The approach to combine the evidence-based medicine and complex systems paradigms is also highly applicable to digital health technologies, given that there is typically an indirect and confounded causal relationship between technology intervention and the desired outcomes. The evidence-based medicine paradigm can be applied to system development and summative evaluation, while implementation and formative evaluation to assess utility and use of digital health technologies needs to be guided by a complex systems paradigm based on natural experiments. During design and system development, developers can use the evidence-based paradigm to examine usability and performance of software as well as discrimination, accuracy, and precision of any embedded algorithms. A wide variety of commercial off-the-shelf solutions means that health systems and organisations are frequently faced with the task of choosing technologies that best meet their needs. Evaluation during selection and implementation should therefore shift to using the complex systems paradigm to assess which technology best fits local requirements. During selection it is also important to evaluate any risks arising from data quality issues, and poor fit of the foundational data to a new situation, such as different population and morbidity patterns. Once a technology is suitably embedded, i.e. its use and utility in a particular context is established using natural experiments, controlled studies including randomized controlled trials can be considered to assess impact on care delivery and patient outcomes. In the next section we show how this combined approach can be applied during the pandemic.

3 Digital Health Evaluation in Context of COVID-19

One way to approach the evaluation of digital health technologies for COVID-19 is based on their technical maturity or their readiness for use in care delivery, as well as the scale of implementation. Three groups of technologies can be considered (Table 1):

1. Mature technologies: These are technologies whose efficacy has been previously demonstrated and whose limitations are known and/or have been addressed. The widespread deployment of telehealth services is perhaps the best example of the effective use of a mature technology. Telehealth services have played a key role in supporting the management of mild COVID-19 at home on a small and large scale [[13]]. Telehealth has also helped to mitigate the impact on delivery of outpatient services that are closed entirely, due to capacity issues or safety concerns. The pandemic has seen health organisations rapidly expand their telehealth capabilities using home grown and commercial platforms. Here pragmatic, rapid evaluation including iterative improvement cycles based on the complex systems paradigm can be used to examine the effectiveness of telehealth as well as any emerging safety concerns in new settings [[14]]. Evaluation can also help identify the contexts of use where telehealth is appropriate, contexts where face-to-face care should be delivered, and contexts where a blended model should be adopted [[15]]. A key research question is how to effectively integrate telehealth as a core-component of a reimagined care process to improve care delivery and patient outcomes [[16]].

   Mature technologies can also be adapted to new uses. For instance, the use of existing national or regional ICU quality registries to provide insight into bed occupancy among hospitals and to calculate the reproduction factor. In the Netherlands, the infrastructure built by the National Intensive Care Evaluation registry allowed the Dutch government to quickly determine the exact number of COVID-19 patients in ICUs [[17]]. This same infrastructure was also quickly adapted and expanded to monitor hospital cases. Such mature technologies do not need extensive evaluation.

2. New technologies deployed on a large scale: The pandemic has encouraged governments to embrace new digital health solutions in an unprecedented manner. Contact tracing apps are an example of a complex technology that has been deployed on a large scale in many nations despite being largely unproven in real-world outbreak settings [[18]]. Issues with their performance as well as security and privacy concerns are useful to illustrate the importance of early evaluation [[19]]. Singapore was the first government to introduce a contact tracing app nationally and reported that it had helped reduce the time taken to identify and quarantine close contacts from four days to two [[20]]. The country went on to introducing Bluetooth contact-tracing tokens in September 2020 to address privacy concerns as well as to cover people without a smart phone, and those who had encountered problems using the app. However, Australia’s COVIDSafe app, which was implemented alongside highly effective manual contact tracing programs, had picked up only 17 positive contacts over 6 months since its launch in late April 2020, at a cost of AUD12 million [[21]].

   For such new technologies, early formative evaluation using an evidence-based medicine paradigm can help to firstly ensure that such apps are usable and are able to provide contact tracers relevant information in a timely manner. In the context of highly effective manual processes, a second stage summative evaluation using the complex systems paradigm can be conducted to examine the value of information provided by an app in improving contact tracing, prior to its widespread deployment [[22]]. In some cases, introduction of an app on a large scale may not be justified if it does not provide useful additional information to improve the number of contacts identified, reduce the time taken to identify contacts or reduce errors. While this may be clear in hindsight, it is an important lesson for future time-sensitive situations. The success of a digital health technology is dependent on the value of information it provides to support care delivery relative to the sociotechnical context of the real-world setting where it is implemented. During a pandemic there is an inherent tension between evaluation and the imperative to urgently deploy solutions which needs to be negotiated.

3. New technologies deployed on a small scale: COVID-19 has sparked local innovation in a wide variety of digital health solutions to support the immediate response to the outbreak as well as mitigating its impact on the delivery of routine health services. For example, coronavirus dashboards give signals when (a region in) a country exceeds thresholds such as number of positive test results or number of ICU or hospital admissions, indicating that more stringent social measures such as lockdowns must be applied [[23]]. Another example comes from US health systems that have leveraged their electronic health records infrastructure to support rapid screening processes, laboratory testing, clinical decision support, reporting tools and patient-facing technology to manage COVID-19 [[24]–[27]]. Evaluation in this context needs to be approached from a complex systems paradigm by necessity as the technology is often discreet and built to rapidly evolving requirements involving multiple stakeholders. Decisions might be driven outside of usual IT/patient safety governance structures and more within leadership in emergency command structures with rapid review from the formal bodies. Pragmatic, rapid cycle and in a way, agile evaluation including iterative improvement cycles in the context of small-scale implementation facilitates adaptation to local needs.

   Another area of local innovation is in computer algorithms for COVID-19 screening. One such algorithm that screened electronic records from telehealth visits to triage patients for viral testing and management, was able to identify important symptoms such as the lack of smell and taste ahead of the US Centers for Disease Control and Prevention guidelines [[28]]. Like other software, evaluation of the performance of computer algorithms needs to be approached from the evidence-based medicine paradigm to assess discrimination, accuracy, and precision prior to their deployment. Care should also be taken to ensure that privacy and security are not compromised in the desire to find solutions quickly [[29]]. The ethical implications of healthcare data outsourcing to corporations should also be considered.

Table 1 Approaches to evaluate different digital health technologies for COVID-19.

Technology type	Evaluation approach	Focus
1. mature technology	complex systems paradigm	- effectiveness in new setting - clinical safety
2. new technology, large scale deployment	evidence-based medicine complex systems paradigm	- usability - performance - value of information
3. new technology, small scale deployment	evidence-based medicine	- usability - performance

4 Conclusions

There is a need to combine the evidence-based medicine and complex systems paradigms to ensure digital health technologies for managing the COVID-19 pandemic are applied safely and effectively. Pragmatic, rapid evaluation using a complex systems paradigm may be more suitable to examine the safety and effectiveness of mature technologies like telehealth in new settings. New technologies, particularly those intended for use on a large scale, will require rigorous assessment of their usability and performance prior to deployment, after which evaluation can shift to using a complex systems paradigm to examine the value of information provided. While it may be practicable to implement and rapidly adapt new discreet technologies to local needs on a small scale, the value of information provided should be demonstrated before proceeding with large scale implementation of complex new technology with distributed effects.

No conflict of interest has been declared by the author(s).

• References

• 1 Mahase E. Covid-19: What do we know about “long covid”?. BMJ 2020; 370
• 2 Grabowski DC, JoyntMaddox KE. Postacute Care Preparedness for COVID-19: Thinking Ahead. JAMA 2020; 323 (20) 2007-8
• 3 Budd J, Miller BS, Manning EM, Lampos V, Zhuang M, Edelstein M. et al. Digital technologies in the public-health response to COVID-19. Nat Med 2020; 26 (08) 1183-92
• 4 Whitelaw S, Mamas MA, Topol E, VanSpall HGCl. Applications of digital technology in COVID-19 pandemic planning and response. Lancet Digit Health 2020; 2 (08) e435-e440
• 5 Li G, Fan G, Chen Y, Deng Z. What patients “see” doctors in online fever clinics during COVID-19 in Wuhan?. J Am Med Inform Assoc 2020; 27 (07) 1067-71
• 6 Yan A, Zou Y, Mirchandani DA. How hospitals in mainland China responded to the outbreak of COVID-19 using information technology–enabled services: An analysis of hospital news webpages. J Am Med Inform Assoc 2020; 27 (07) 991-9
• 7 AlKnawy B, Adil M, Crooks G, Rhee K, Bates D, Jokhdar H. et al. The Riyadh Declaration: the role of digital health in fighting pandemics. Lancet 2020; 396 (10262): 1537-9
• 8 Gerke S, Shachar C, Chai PR, Coehn IG. Regulatory, safety, and privacy concerns of home monitoring technologies during COVID-19. Nat Med 2020; 26 (08) 1176-82
• 9 Ammenwerth E, Rigby M. Evidence-Based Health Informatics: Promoting Safety and Efficiency Through Scientific Methods and Ethical Policy. IOS Press; 2016
• 10 Coiera E. Guide to health informatics, third edition. CRC Press; 2015
• 11 Greenhalgh T. Will COVID-19 be evidence-based medicine’s nemesis?. PLoS Med 2020; 17 (06) e1003266
• 12 Cresswell K, Williams R, Sheikh A. Developing and Applying a Formative Evaluation Framework for Health Information Technology Implementations: Qualitative Investigation. J Med Internet Res 2020; 22 (06) e15068
• 13 Wosik J, Fudim M, Cameron B, Gellad ZF, Cho A, Phinney D. et al. Telehealth transformation: COVID-19 and the rise of virtual care. J Am Med Inform Assoc 2020; 27 (06) 957-62
• 14 Lieneck CAO, Garvey J, Collins C, Graham D, Lovin C, Pearson R. Rapid Telehealth Implementation during the COVID-19 Global Pandemic: A Rapid Review. Healthcare (Basel) 2020; 8 (04) 517
• 15 Kaplan B. Revisiting health information technology ethical, legal, and social issues and evaluation: telehealth/telemedicine and covid-19. Int J Med Inform 2020; 143: 104239
• 16 Dorn SD. Backslide or forward progress? Virtual care at U.S. healthcare systems beyond the COVID-19 pandemic. NPJ Digit Med 2021; 4 (01) 6
• 17 RIVM, the Dutch National Institute for Public Health and the Environment. Available from: https://www.rivm.nl/en/news/slight-decrease-in-number-of-new-covid-19-infections-reported
• 18 Anglemyer A, Moore TH, Parker L, Chambers T, Grady A, Chiu K. et al. Digital contact tracing technologies in epidemics: a rapid review. Cochrane Database Syst Rev 2020; 8: Cd013699
• 19 Ahmed N, Michelin RA, Xue W, Ruj S, Malaney R, Kanhere SS. et al. A Survey of COVID-19 Contact Tracing Apps. IEEE Access 2020; 8: 134577-601
• 20 Singapore distributes Covid contact-tracing tokens. BBC News 2020 14 Sep. Available from: https://www.bbc.com/news/business-54143015
• 21 McDonald K. COVIDSafe app criticised for high cost, low effectiveness. Pulse+IT [Internet]. 2020 [cited 2021 18 Feb]; (28 Oct). Available from: https://www.pulseitmagazine.com.au/australian-ehealth/5782-covidsafe-app-criticised-for-high-cost-low-effectiveness
• 22 Coiera E. Assessing Technology Success and Failure Using Information Value Chain Theory. Stud Health Technol Inform 2019; 263: 35-48
• 23 Coronavirus dashboard: Ministry of Health, Welfare and Sport, the Netherlands. Available from: https://coronadashboard.government.nl/
• 24 Dixit RA, Hurst S, Adams KT, Boxley C, Lyen-Hendershot K, Bennett SS. et al. Rapid development of visualization dashboards to enhance situation awareness of COVID-19 telehealth initiatives at a multihospital healthcare system. J Am Med Inform Assoc 2020; 27 (09) 1456-61
• 25 Ford D, Harvey JB, McElligott J, King K, Simpson KN, Valenta S. et al. Leveraging health system telehealth and informatics infrastructure to create a continuum of services for COVID-19 screening, testing, and treatment. J Am Med Inform Assoc 2020; 27 (12) 1871-7
• 26 Judson TJ, Odisho AY, Neinstein AB, Chao J, Williams A, Miller C. et al. Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19. J Am Med Inform Assoc 2020; 27 (06) 860-6
• 27 Reeves JJ, Hollandsworth HM, Torriani FJ, Tapliz R, Abeles S, Tai-Seale M. et al. Rapid response to COVID-19: health informatics support for outbreak management in an academic health system. J Am Med Inform Assoc 2020; 27 (06) 853-9
• 28 Obeid JS, Davis M, Turner M, Meystre SM, Heider PM, O’Bryan EC. et al. An artificial intelligence approach to COVID-19 infection risk assessment in virtual visits: A case report. J Am Med Inform Assoc 2020; 27 (08) 1321-5
• 29 Secret data and the future of public health: why the NHS has turned to Palantir. New Statesman 2020 21 May. Available from: https://www.newstatesman.com/politics/health/2020/05/secret-data-and-future-public-health-why-nhs-has-turned-palantir

Publication History

Article published online:
21 April 2021