CC BY-NC-ND 4.0 · Yearb Med Inform 2021; 30(01): 056-060
DOI: 10.1055/s-0041-1726490
Special Section: Managing Pandemics with Health Informatics
Working Group Contributions

Managing Pandemic Responses with Health Informatics – Challenges for Assessing Digital Health Technologies

A Joint Position Paper from the IMIA Technology Assessment & Quality Development in Health Informatics Working Group and EFMI Working Group for Assessment of Health Information Systems
Farah Magrabi
1   Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
Elske Ammenwerth
2   UMIT, Private University for Health Sciences, Medical Informatics and Technology, Institute of Medical Informatics, Hall in Tirol, Austria
Catherine K. Craven
3   Institute for Health Care Delivery Science, Icahn School of Medicine at Mount Sinai, New York, New York, USA
Kathrin Cresswell
4   Usher Institute, The University of Edinburgh, United Kingdom
Nicolet F. De Keizer
5   Amsterdam UMC, University of Amsterdam, Department of Medical Informatics, Amsterdam Public Health research institute, The Netherlands
Stephanie K. Medlock
5   Amsterdam UMC, University of Amsterdam, Department of Medical Informatics, Amsterdam Public Health research institute, The Netherlands
Philip J. Scott
6   University of Portsmouth, Centre for Healthcare Modelling and Informatics, Portsmouth, United Kingdom
Zoie Shui-Yee Wong
7   Graduate School of Public Health, St. Luke’s International University, Tokyo, Japan
Andrew Georgiou
1   Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
› Author Affiliations


Objectives: To highlight the role of technology assessment in the management of the COVID-19 pandemic.

Method: An overview of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems.

Results: Evaluation of digital health technologies for COVID-19 should be based on their technical maturity as well as the scale of implementation. For mature technologies like telehealth whose efficacy has been previously demonstrated, pragmatic, rapid evaluation using the complex systems paradigm which accounts for multiple sociotechnical factors, might be more suitable to examine their effectiveness and emerging safety concerns in new settings. New technologies, particularly those intended for use on a large scale such as digital contract tracing, will require assessment of their usability as well as performance prior to deployment, after which evaluation should shift to using a complex systems paradigm to examine the value of information provided. The success of a digital health technology is dependent on the value of information it provides relative to the sociotechnical context of the setting where it is implemented.

Conclusion: Commitment to evaluation using the evidence-based medicine and complex systems paradigms will be critical to ensuring safe and effective use of digital health technologies for COVID-19 and future pandemics. There is an inherent tension between evaluation and the imperative to urgently deploy solutions that needs to be negotiated.

Publication History

Article published online:
21 April 2021

© 2021. IMIA and Thieme. This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Mahase E. Covid-19: What do we know about “long covid”?. BMJ 2020; 370
  • 2 Grabowski DC, JoyntMaddox KE. Postacute Care Preparedness for COVID-19: Thinking Ahead. JAMA 2020; 323 (20) 2007-8
  • 3 Budd J, Miller BS, Manning EM, Lampos V, Zhuang M, Edelstein M. et al. Digital technologies in the public-health response to COVID-19. Nat Med 2020; 26 (08) 1183-92
  • 4 Whitelaw S, Mamas MA, Topol E, VanSpall HGCl. Applications of digital technology in COVID-19 pandemic planning and response. Lancet Digit Health 2020; 2 (08) e435-e440
  • 5 Li G, Fan G, Chen Y, Deng Z. What patients “see” doctors in online fever clinics during COVID-19 in Wuhan?. J Am Med Inform Assoc 2020; 27 (07) 1067-71
  • 6 Yan A, Zou Y, Mirchandani DA. How hospitals in mainland China responded to the outbreak of COVID-19 using information technology–enabled services: An analysis of hospital news webpages. J Am Med Inform Assoc 2020; 27 (07) 991-9
  • 7 AlKnawy B, Adil M, Crooks G, Rhee K, Bates D, Jokhdar H. et al. The Riyadh Declaration: the role of digital health in fighting pandemics. Lancet 2020; 396 (10262): 1537-9
  • 8 Gerke S, Shachar C, Chai PR, Coehn IG. Regulatory, safety, and privacy concerns of home monitoring technologies during COVID-19. Nat Med 2020; 26 (08) 1176-82
  • 9 Ammenwerth E, Rigby M. Evidence-Based Health Informatics: Promoting Safety and Efficiency Through Scientific Methods and Ethical Policy. IOS Press; 2016
  • 10 Coiera E. Guide to health informatics, third edition. CRC Press; 2015
  • 11 Greenhalgh T. Will COVID-19 be evidence-based medicine’s nemesis?. PLoS Med 2020; 17 (06) e1003266
  • 12 Cresswell K, Williams R, Sheikh A. Developing and Applying a Formative Evaluation Framework for Health Information Technology Implementations: Qualitative Investigation. J Med Internet Res 2020; 22 (06) e15068
  • 13 Wosik J, Fudim M, Cameron B, Gellad ZF, Cho A, Phinney D. et al. Telehealth transformation: COVID-19 and the rise of virtual care. J Am Med Inform Assoc 2020; 27 (06) 957-62
  • 14 Lieneck CAO, Garvey J, Collins C, Graham D, Lovin C, Pearson R. Rapid Telehealth Implementation during the COVID-19 Global Pandemic: A Rapid Review. Healthcare (Basel) 2020; 8 (04) 517
  • 15 Kaplan B. Revisiting health information technology ethical, legal, and social issues and evaluation: telehealth/telemedicine and covid-19. Int J Med Inform 2020; 143: 104239
  • 16 Dorn SD. Backslide or forward progress? Virtual care at U.S. healthcare systems beyond the COVID-19 pandemic. NPJ Digit Med 2021; 4 (01) 6
  • 17 RIVM, the Dutch National Institute for Public Health and the Environment. Available from:
  • 18 Anglemyer A, Moore TH, Parker L, Chambers T, Grady A, Chiu K. et al. Digital contact tracing technologies in epidemics: a rapid review. Cochrane Database Syst Rev 2020; 8: Cd013699
  • 19 Ahmed N, Michelin RA, Xue W, Ruj S, Malaney R, Kanhere SS. et al. A Survey of COVID-19 Contact Tracing Apps. IEEE Access 2020; 8: 134577-601
  • 20 Singapore distributes Covid contact-tracing tokens. BBC News 2020 14 Sep. Available from:
  • 21 McDonald K. COVIDSafe app criticised for high cost, low effectiveness. Pulse+IT [Internet]. 2020 [cited 2021 18 Feb]; (28 Oct). Available from:
  • 22 Coiera E. Assessing Technology Success and Failure Using Information Value Chain Theory. Stud Health Technol Inform 2019; 263: 35-48
  • 23 Coronavirus dashboard: Ministry of Health, Welfare and Sport, the Netherlands. Available from:
  • 24 Dixit RA, Hurst S, Adams KT, Boxley C, Lyen-Hendershot K, Bennett SS. et al. Rapid development of visualization dashboards to enhance situation awareness of COVID-19 telehealth initiatives at a multihospital healthcare system. J Am Med Inform Assoc 2020; 27 (09) 1456-61
  • 25 Ford D, Harvey JB, McElligott J, King K, Simpson KN, Valenta S. et al. Leveraging health system telehealth and informatics infrastructure to create a continuum of services for COVID-19 screening, testing, and treatment. J Am Med Inform Assoc 2020; 27 (12) 1871-7
  • 26 Judson TJ, Odisho AY, Neinstein AB, Chao J, Williams A, Miller C. et al. Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19. J Am Med Inform Assoc 2020; 27 (06) 860-6
  • 27 Reeves JJ, Hollandsworth HM, Torriani FJ, Tapliz R, Abeles S, Tai-Seale M. et al. Rapid response to COVID-19: health informatics support for outbreak management in an academic health system. J Am Med Inform Assoc 2020; 27 (06) 853-9
  • 28 Obeid JS, Davis M, Turner M, Meystre SM, Heider PM, O’Bryan EC. et al. An artificial intelligence approach to COVID-19 infection risk assessment in virtual visits: A case report. J Am Med Inform Assoc 2020; 27 (08) 1321-5
  • 29 Secret data and the future of public health: why the NHS has turned to Palantir. New Statesman 2020 21 May. Available from: