Subscribe to RSS
DOI: 10.1055/s-0041-1726524
Human Factors and Organizational Issues Section Synopsis IMIA Yearbook 2021
Summary
Objective: To select the best papers that made original and high impact contributions in the area of human factors and organizational issues in biomedical informatics in 2020.
Methods: A rigorous extraction process based on queries from Web of Science® and PubMed/Medline was conducted to identify the scientific contributions published in 2020 that address human factors and organizational issues in biomedical informatics. The screening of papers on titles and abstracts independently by the two section editors led to a total of 1,562 papers. These papers were discussed for a selection of 12 finalist papers, which were then reviewed by the two section editors, two chief editors, and by three external reviewers from internationally renowned research teams.
Results: The query process resulted in 12 papers that reveal interesting and rigorous methods and important studies in human factors that move the field forward, particularly in clinical informatics and emerging technologies such as brain-computer interfaces. This year three papers were clearly outstanding and help advance in the field. They provide examples of applying existing frameworks together in novel and highly illuminating ways, showing the value of theory development in human factors. Emerging themes included several which discussed physician burnout, mobile health, and health equity. Those concerning the Corona Virus Disease 2019 (Covid-19) were included as part of that section.
Conclusion: The selected papers make important contributions to human factors and organizational issues, expanding and deepening our knowledge of how to apply theory and applications of new technologies in health.
#
Keywords
Human factors - human-computer interaction - organizational issues - electronic health records - mobile health1 Introduction
Papers published in 2020 on human factors and organizational issues (HFOI) focused on user healthcare technology and data needs, for both patients and physicians. In addition, several papers focused on the downstream impact of Electronic Health Record (EHR) use and usability on provider burnout. Not surprisingly, there were also many papers about designing and implementing new systems for Coronavirus Disease 2019 (COVID-19), as well as on design considerations for achieving health equity, both very timely topics in 2020. Finally, there were other papers that focused on design factors for healthcare systems. Together these papers provide a small cross section of the best HFOI research in biomedical informatics in 2020.
#
2 Methods
We collected papers from 2020 by a standard search in PubMed and Web of Science®. Searches were performed in January 2020 to identify peer-reviewed journal articles published in the English language in 2020, and related to HFOI research in biomedical informatics. In addition to the search of electronic databases, manual searches of key themes were performed in major biomedical journals (e.g., Journal of the American Medical Informatics Association, Methods of Information in Medicine, Journal of Medical Internet Research, …etc.). Independent searches by the two co-editors resulted in an initial set of 1,562 papers. Independent reading of the titles and abstracts resulted in each co-editor creating a set of 15 candidates each, which were then discussed. From this search we selected 13 best papers; these were sent for additional peer review by a set of external reviewers and two chief editors, with each paper being reviewed by at least four people. We categorized the best papers according to four themes: 1) data presentation for accuracy and understanding; 2) EHR usability and clinician burnout; 3) clinical system design; and 4) timely topics. We then selected the best four papers using these reviews, while also ensuring the papers represented a variety of themes and had strong methods and results.
#
3 Results
[Table 1] lists the top four papers, in order of the first author's surname.
Adler-Milstein et al. [[1]] combined common objective measures of EHR use with Maslach Burnout inventory survey responses for 87 clinicians in primary care clinics of a large, urban, academic medical center to determine how EHR use might be affecting burnout. They found that 34% of clinicians had high cynicism and 51% had high emotional exhaustion. Clinicians who had the highest measures of EHR time after hours had much higher odds of high exhaustion; this trend also held for clinicians with the most message volume. No trends were observed for cynicism.
In their paper, Brewer et al. [[2]] present contextually tailored case examples of studies and projects which aim to address health inequities by involving the community, i.e., co-design approaches. It strategically merged health services research with community-based participatory research (CBPR) for innovation and development. The primary contribution of this paper is a set of resilience strategies that could be supported by innovations in health information technology in future research. The paper addresses the serious problem of health equity and calls attention to the possibility that the new massive activity in health informatics including digital health may have an unintended consequences: to worsen health inequities and create new forms of inequity. Thus, designers and informaticians must guard against these consequences in their ongoing work.
Patient-reported outcomes (PROs) increasingly use self-monitor health status and visualization over time would help patients interpret their data. The study of Reading Turchioe et al. [[3]] included 40 hospitalized patients and compared four visualization conditions: (1) text-only; (2) text plus visual analogy; (3) text plus number line; and (4) text plus line graph assessed objective comprehension using the International Organization for Standardization protocol. Secondary outcomes included response times, preferences, risk perceptions, and behavioral intentions. As a result, 63% correctly comprehended the text-only condition and 60% comprehended the line graph condition, compared with 83% for the visual analogy and 70% for the number line (P< 0.05) conditions. This was the top voted paper and provided the best example of a continued emerging emphasis on visualization for patients, including those with lower literacy, and cautioned against over-reliance on graphs.
Finally, Tschandl et al. [[4]] tested a variety of representations of artificial intelligence (AI) for rating skin lesions for providing decision support to physicians and measured the resulting accuracy and susceptibility to randomly introduced errors. They found that when they provided the probabilities for the full range of diagnoses, the decision support improved the physician's diagnostic accuracy by 13% but that the decision support for determination of malignancy did not improve accuracy. They also found that physicians were susceptible to accepting randomly incorrect predictions once they had used the AI based decision support tools.
Although not selected as best papers, the remaining candidate best papers were just as excellent and influential. We summarize them below, organized by the four major themes for the papers: 1) data presentation for accuracy and understanding; 2) EHR usability and clinician burnout; 3) clinical system design; and 4) timely topics.
3.1 Data Presentation for Accuracy and Understanding
Two of the four selected best papers fell into this category; the Reading-Turchioe et al. paper [[3]] focused on presenting healthcare data to patients while the Tschandl et al. [[4]] paper focused on presenting results from AI models for diagnosing skin lesions. In addition, Stonbraker et al. [[5]] designed a thorough study of data presentations of longitudinal patient reported outcomes to determine which ones promoted the best comprehension. They found that patients interpreted the visualizations very literally and that while they preferred bar charts with emojis, they could confuse the meaning of the emojis if they were not precisely tailored to the symptom or outcome they were representing.
#
3.2 EHR Usability and Clinician Burnout
The Adler-Milstein paper et al. paper [[1]] described above fell into this category. Another excellent paper by Melnick et al. [[6]] performed a secondary analysis of a survey of US physicians from all specialties to determine the association between physicians’ perceived EHR usability and workload and their professional burnout. They found a strong relationship between EHR usability and workload, with more favorable usability associated with a lower task load.
Another innovative paper by Weir et al. [[7]] was a position paper that explored how burnout is affected by three aspects of cognition and motivation: 1) goal-based decision making; 2) drive for sense-making and meaning; and 3) need for agency, autonomy, and control. Through a thorough discussion of prior research, the paper hoped to spur research that applies concepts of cognitive science to understanding and mitigating physician burnout.
#
3.3 Clinical Systems Design
This theme had the broadest topics of papers, from user and patient designed systems, to the negative impact of a medication ordering system, to the state of application programming interfaces in healthcare. A paper by Farao et al. [[8]] described a user-centered design framework for mobile health applications and demonstrated its use for an app that read the tuberculin skin test. Combining the information systems research framework with design thinking, the new framework allowed for iterative refinement of designs through interactions with users. A paper by Ng et al. [[9]] described patient designed enhancements to a continuous glucose monitoring system for Type I diabetes. These “do-it-yourself” innovations formed the Nightscout Project, a continually expanding set of open-source technologies driven by a community of Type I diabetic patients, their parents, and their caregivers. Next, a paper by MacKenna et al. [[10]] described how a poorly designed medication ordering system resulted in 9.5 million excess cost in one year for ordering more expensive generic medication than was necessary. Finally, a paper by Dullabh et al. [[11]] used a socio-technical model to structure an analysis of the current state of application programming interfaces in healthcare for providing necessary interoperability and data sharing. Not surprisingly, they found a large amount of potential use cases and existing applications, most of which focused on filling a physician's need, read capabilities were widely provided, but not write, FHIR was the overwhelming standard that is used, and it enabled communication of the common clinical dataset, and validating data from external sources complicated providers’ workflows.
#
3.4 Timely Topics
This final theme included timely topics from 2020: COVID-19 and health equity. One of the best papers, Brewer et al. [[2]] demonstrated how user-centered design can have a big impact on addressing health equity. Two other papers were on COVID-19. Reeves et al. [[12]] described the design and implementation of EHR based tools for managing multiple aspects of COVID-19 and is included in the special section of the yearbook. Another paper by Altmann et al. [[13]] employed a survey across five countries to determine the acceptability of an automated contact tracing application. They found widespread support for such an application and concluded it could be a viable tool for limiting spread of the virus.
#
#
4 Conclusion
The HFOI papers this year reflected important progress and diversification of themes. The pandemic and issues it has raised formed an important set of advances, but also expanded existing topics such as clinician burnout and the importance of usability in patient-facing interventions. The emergence of equity and usability as a serious concern is overdue, and all these advances are welcome expansions in the field.
#
Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K
Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians
J Am Med Inform Assoc 2020;27(4):531-8
Primary care physicians have among the highest rate of burnout of all clinical specialties; further, these physicians have self-reported after hours work as a key factor for feelings of burnout. This study compared self-reported and objective measures of electronic health record (EHR) work and proficiency and then correlated them with burnout measures for primary care providers to better understand and improve factors driving burnout. To do this, the study used self-reported measures of EHR use and proficiency, EHR supplied metrics for after-hours EHR use, messaging volume, and proficiency, and Maslach Burnout Inventory exhaustion and cynicism subscale responses for 87 primary care providers in an urban academic medical center. The results showed that the subjective, self-reported perceived EHR use time was correlated with objective measures of EHR use after-hours and on unscheduled days, message volume, and proficiency metrics. There was not a relationship between the subjective and objective measures of EHR proficiency. About one-third (34%) of providers reported high cynicism and 51% reported high emotional exhaustion. Those providers in the top 2 quartiles of EHR use after hours had 4.78 (95% confidence interval [CI], 1.1-20.1; P=0.04) and 12.52 (95% CI, 2.6-61; P=0.002) greater odds of high exhaustions. Similarly, clinicians in the top quartile of message volume had 6.17 greater odds of high exhaustion (95% CI, 1.1-41; P = .04). No objective measures were associated with high cynicism. These results are important since they fill an important gap of correlating self-reported EHR use with widely reported objective EHR metrics of use, as well as correlating objective measures of EHR use with components of burnout. Since prior studies have correlated self-reported EHR use time with burnout, there was a need to validate this correlation with objective measures of use time, particularly as interventions are designed to address EHR burden. Interestingly, after hours EHR use was associated with exhaustion, but not cynicism, suggesting that providers were feeling overwhelmed by work but did not develop cynicism as a response. The authors were hopeful that interventions designed to reduce the EHR burden (i.e., after hours work and message volume) could help reduce these providers’ exhaustion. It is possible, however, that providers who have developed cynicism are those who have potentially inappropriately cut short their EHR use as a coping mechanism. Further studies are needed to further illuminate the relationship between cynicism and EHR burden in order to design effective interventions.
Brewer LC, Fortuna KL, Jones C, Walker R, Hayes SN, Patten CA, Cooper LA
Back to the future: Achieving health equity through health informatics and digital health
JMIR Mhealth Uhealth 2020;8(1):e14512
This paper called attention to the serious issues of health equity and its systemic effects and unintended consequences. These issues were brought to the fore during the pandemic, with some technology advances and uses creating further inequities that need to be addressed. The authors presented a set of principles and two example projects, contextually tailored, which aimed to address health inequities by community involvement, co-design approaches, and strategically merged health services research with community-based participatory research for innovation and development. The primary contribution of the paper was a set of resilience strategies that could be supported by health information technology in future work. The first project, FAITH! (Fostering African American Improvement in Total Health), described an intervention based on mHealth and face to face church-based health education and social support, especially directed at cardiovascular disease which has double the mortality rate in African Americans compared to whites, and a higher incidence. They used iterative formative design processes with a team of clinicians, technologists, behavioral and social scientists, and made use of patient preferences, such as using spiritual verses and messaging. This led to high apps ratings and acceptability, usability, and satisfaction. Blood pressure, diet, and physical activity all significantly improved, and the app had a 98% retention rate, remarkable for a health app. Tailored visual study results were also fed back to the community. The second project, Peertech CBPR partnership, addressed premature mortality in people with serious mental illness (SMI) sch as bipolar disorder, major depressive disorder, and schizophrenia. The partnering academic-community team identified this as a major health disparity. This led to co-creation of a smartphone app: Peer- and Technology-Supported Self-Management Training (PeerTECH), which carried out simultaneous management of mental and chronic health conditions in patients aged over 60. The approach consisted of equal partnership between patients, certified peer specialists (CPS), leaders, and scientists from idea conception, defining of research questions, intervention development and usability testing extending to dissemination. Results included statistically significant psychiatric self-management (on the Illness Management and Recovery Scale (IMRS) (p<0.001) and improvements in medical self-management, hope, quality of life, and empowerment. The authors summarized best practices for strategic design and implementation of digital health interventions for the marginalized:
-
Increase recruitment and retention of diverse populations throughout R&D; assess differential responses/outcomes of technologies; mitigate preferential access;
-
Leverage established stakeholders and trusted social networks;
-
Understand the social context of potential end users and populations;
-
Integrate community engagement through user-centered design or participatory design;
-
Gain an understanding of community partner technology infrastructure for capacity building to support and strengthen community-based health informatics interventions; and
-
Plan the appropriate amount of time and resources to devote to community engagement processes for intervention development and sustainability.
The importance of this paper is in addressing a current timely issue, with an unprecedented degree of success in retention and disease control effectiveness, serving as an example of how to conduct such design successfully.
Reading Turchioe M, Grossman LV, Myers AC, Baik D, Goyal P, Masterson Creber RM
Visual analogies, not graphs, increase patients’ comprehension of changes in their health status
J Am Med Inform Assoc 2020;27(5):677-89
The aim of this paper was to understand which visualizations are most effective in aiding patients to self-monitor their health status, via the use of patient-reported outcome (PRO) measures in hospitalized patients, with a focus on objective comprehension. 40 hospitalized patients were included to compare fourvisualization conditions: (1) text-only, (2) text plus visual analogy, (3) text plus number line, and (4) text plus line graph assessed objective comprehension using the International Organization for Standardization protocol. Secondary outcomes included response times, preferences, risk perceptions, and behavioral intentions. Sixty-three percent correctly comprehended the text-only condition and 60% comprehended the line graph condition, compared with 83% for the visual analogy and 70% for the number line (P< 0.05) conditions. The results supported using visual analogies rather than text to display longitudinal PROs but the authors cautioned against relying on graphs (known high prevalence of inadequate graph literacy). Discrepancies between comprehension and preferences suggested factors other than comprehension influenced preferences. Future researchers should assess comprehension rather than preferences to guide presentation decisions. This paper was the best example of a continued emerging emphasis on visualization for patients and clinicians to improve comprehension and workflow.
Tschandl P, Rinner C, Apalla Z, Argenziano G, Codella N, Halpern A, Janda M, Lallas A, Longo C, Josep Malvehy J, Paoli J, Puig S, Rosendahl C, Soyer HP, Zalaudek I, Kittler H
Human–computer collaboration for skin cancer recognition
Nat Med 2020;26(8):1229–34
Image-based artificial intelligence (AI) has the potential to improve visual diagnostic accuracy in healthcare. Recent studies in dermatology have shown that the accuracy of AI for identifying skin lesions was equivalent to or better than human experts in controlled experimental studies; further, human-AI cooperation can improve accuracy even more, but studies have not determined the best ways to incorporate AI into clinical workflows for improving diagnostic accuracy in real healthcare settings. This study explored the impact of different representations of AI based clinical decision support for identifying skin lesions on the accuracy of clinical diagnoses. The first representation presented the probability that the lesion was in one of seven diagnostic categories: four malignant and three benign. The second collapsed the seven categories into two: malignant and benign. The third used image retrieval to present similar images with known diagnoses and the fourth presented previously collected probabilities for each of the seven diagnoses as determined by 511 human raters. A total of 302 human raters from 41 countries diagnosed batches of images both without decision support and then with one type of decision support. The results showed that the first representation improved the accuracy of human raters from 63.6% to 77.0% (increase of 13.3%, 95% CI 11.5% to 15.2%; P = 4.9 × 10−35, two-sided paired t-test, t = 14.5, d.f. = 301; n = 302 raters). No improvement was observed for the second or third representation, with some improvement for the fourth. They observed an inverse relationship between the net gain accuracy increase due to AI and rater experience; more inexperienced raters more frequently changed their initial ratings after viewing the AI results than experts did. After establishing a positive impact of AI on diagnostic accuracy, the study tested the impact of errors in AI on diagnostic accuracy. These errors could be due to models classifying images outside the domain of images used to train the model, something which is a big concern for AI researchers. They intentionally showed raters incorrect multiclass probabilities, which decreased the providers’ accuracy back to where it was without AI. The paper was very thorough and presented several other analyses such as identifying two different diagnostic categories that benefitted most of clinical decision support based on AI, measuring the tendency of raters to change their minds, and determining how background features affected both the AI and the human rater's classification. Based on these multiple analyses, this paper justifiably advocates for studying human computer collaboration in real world clinic settings when evaluating the performance of AI for any and all healthcare applications, not just those in dermatology.
#
No conflict of interest has been declared by the author(s).
Acknowledgements
We would like to acknowledge the support of Adrien Ugon, Martina Hutter, Kate Fultz Hollis, Lina Soualmia, Brigitte Séroussi, and the whole Yearbook Editorial Committee as well as the reviewers for their contribution to the selection process of the Human Factor and Organizational Issues section for the IMIA Yearbook 2021.
-
References
- 1 Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020 Apr 1 27 (04) 531-8
- 2 Brewer LC, Fortuna KL, Jones C, Walker R, Hayes SN, Patten CA. et al. Back to the Future: Achieving Health Equity Through Health Informatics and Digital Health. JMIR MHealth UHealth 2020 Jan 14 8 (01) e14512
- 3 Reading Turchioe M, Grossman LV, Myers AC, Baik D, Goyal P, Masterson Creber RM. Visual analogies, not graphs, increase patients’ comprehension of changes in their health status. J Am Med Inform Assoc 2020 May 1 27 (05) 677-89
- 4 Tschandl P, Rinner C, Apalla Z, Argenziano G, Codella N, Halpern A. et al. Human-computer collaboration for skin cancer recognition. Nat Med 2020 Aug 26 (08) 1229-34
- 5 Stonbraker S, Porras T, Schnall R. Patient preferences for visualization of longitudinal patient-reported outcomes data. J Am Med Inform Assoc 2020 Feb 1 27 (02) 212-24
- 6 Melnick ER, Harry E, Sinsky CA, Dyrbye LN, Wang H, Trockel MT. et al. Perceived Electronic Health Record Usability as a Predictor of Task Load and Burnout Among US Physicians: Mediation Analysis. J Med Internet Res 2020 Dec 22 22 (12) e23382
- 7 Weir CR, Taber P, Taft T, Reese TJ, Jones B, Del Fiol G. Feeling and thinking: can theories of human motivation explain how EHR design impacts clinician burnout?. J Am Med Inform Assoc 2021 Apr 23 28 (05) 1042-6
- 8 Farao J, Malila B, Conrad N, Mutsvangwa T, Rangaka MX, Douglas TS. A user-centred design framework for mHealth. PLOS One 2020 Aug 19 15 (08) e0237910
- 9 Ng M, Borst E, Garrity A, Hirschfeld E, Lee J. Evolution of Do-It-Yourself Remote Monitoring Technology for Type 1 Diabetes. J Diabetes Sci Technol 2020 Sep 14 (05) 854-9
- 10 MacKenna B, Curtis HJ, Walker AJ, Bacon S, Croker R, Goldacre B. Suboptimal prescribing behaviour associated with clinical software design features: a retrospective cohort study in English NHS primary care. Br J Gen Pract J R Coll Gen Pract 2020 Sep 70 (698) e636-43
- 11 Dullabh P, Hovey L, Heaney-Huls K, Rajendran N, Wright A, Sittig DF. Application Programming Interfaces in Health Care: Findings from a Current-State Sociotechnical Assessment. Appl Clin Inform 2020 Jan 11 (01) 59-69
- 12 Reeves JJ, Hollandsworth HM, Torriani FJ, Taplitz R, Abeles S, Tai-Seale M. et al. Rapid response to COVID-19: health informatics support for outbreak management in an academic health system. J Am Med Inform Assoc 2020 Jun 1 27 (06) 853-9
- 13 Altmann S, Milsom L, Zillessen H, Blasone R, Gerdon F, Bach R. et al. Acceptability of App-Based Contact Tracing for COVID-19: Cross-Country Survey Study. JMIR MHealth UHealth [Internet]. 2020 Aug 28 [cited 2021 May 24];8(8). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7458659/
Correspondence to
Publication History
Article published online:
03 September 2021
© 2021. IMIA and Thieme. This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)
Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany
-
References
- 1 Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020 Apr 1 27 (04) 531-8
- 2 Brewer LC, Fortuna KL, Jones C, Walker R, Hayes SN, Patten CA. et al. Back to the Future: Achieving Health Equity Through Health Informatics and Digital Health. JMIR MHealth UHealth 2020 Jan 14 8 (01) e14512
- 3 Reading Turchioe M, Grossman LV, Myers AC, Baik D, Goyal P, Masterson Creber RM. Visual analogies, not graphs, increase patients’ comprehension of changes in their health status. J Am Med Inform Assoc 2020 May 1 27 (05) 677-89
- 4 Tschandl P, Rinner C, Apalla Z, Argenziano G, Codella N, Halpern A. et al. Human-computer collaboration for skin cancer recognition. Nat Med 2020 Aug 26 (08) 1229-34
- 5 Stonbraker S, Porras T, Schnall R. Patient preferences for visualization of longitudinal patient-reported outcomes data. J Am Med Inform Assoc 2020 Feb 1 27 (02) 212-24
- 6 Melnick ER, Harry E, Sinsky CA, Dyrbye LN, Wang H, Trockel MT. et al. Perceived Electronic Health Record Usability as a Predictor of Task Load and Burnout Among US Physicians: Mediation Analysis. J Med Internet Res 2020 Dec 22 22 (12) e23382
- 7 Weir CR, Taber P, Taft T, Reese TJ, Jones B, Del Fiol G. Feeling and thinking: can theories of human motivation explain how EHR design impacts clinician burnout?. J Am Med Inform Assoc 2021 Apr 23 28 (05) 1042-6
- 8 Farao J, Malila B, Conrad N, Mutsvangwa T, Rangaka MX, Douglas TS. A user-centred design framework for mHealth. PLOS One 2020 Aug 19 15 (08) e0237910
- 9 Ng M, Borst E, Garrity A, Hirschfeld E, Lee J. Evolution of Do-It-Yourself Remote Monitoring Technology for Type 1 Diabetes. J Diabetes Sci Technol 2020 Sep 14 (05) 854-9
- 10 MacKenna B, Curtis HJ, Walker AJ, Bacon S, Croker R, Goldacre B. Suboptimal prescribing behaviour associated with clinical software design features: a retrospective cohort study in English NHS primary care. Br J Gen Pract J R Coll Gen Pract 2020 Sep 70 (698) e636-43
- 11 Dullabh P, Hovey L, Heaney-Huls K, Rajendran N, Wright A, Sittig DF. Application Programming Interfaces in Health Care: Findings from a Current-State Sociotechnical Assessment. Appl Clin Inform 2020 Jan 11 (01) 59-69
- 12 Reeves JJ, Hollandsworth HM, Torriani FJ, Taplitz R, Abeles S, Tai-Seale M. et al. Rapid response to COVID-19: health informatics support for outbreak management in an academic health system. J Am Med Inform Assoc 2020 Jun 1 27 (06) 853-9
- 13 Altmann S, Milsom L, Zillessen H, Blasone R, Gerdon F, Bach R. et al. Acceptability of App-Based Contact Tracing for COVID-19: Cross-Country Survey Study. JMIR MHealth UHealth [Internet]. 2020 Aug 28 [cited 2021 May 24];8(8). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7458659/