Review of Clinical Research Informatics

Summary

Objectives: Clinical Research Informatics (CRI) declares its scope in its name, but its content, both in terms of the clinical research it supports—and sometimes initiates—and the methods it has developed over time, reach much further than the name suggests. The goal of this review is to celebrate the extraordinary diversity of activity and of results, not as a prize-giving pageant, but in recognition of the field, the community that both serves and is sustained by it, and of its interdisciplinarity and its international dimension.

Methods: Beyond personal awareness of a range of work commensurate with the author’s own research, it is clear that, even with a thorough literature search, a comprehensive review is impossible. Moreover, the field has grown and subdivided to an extent that makes it very hard for one individual to be familiar with every branch or with more than a few branches in any depth. A literature survey was conducted that focused on informatics-related terms in the general biomedical and healthcare literature, and specific concerns (“artificial intelligence”, “data models”, “analytics”, etc.) in the biomedical informatics (BMI) literature. In addition to a selection from the results from these searches, suggestive references within them were also considered.

Results: The substantive sections of the paper—Artificial Intelligence, Machine Learning, and “Big Data” Analytics; Common Data Models, Data Quality, and Standards; Phenotyping and Cohort Discovery; Privacy: Deidentification, Distributed Computation, Blockchain; Causal Inference and Real-World Evidence—provide broad coverage of these active research areas, with, no doubt, a bias towards this reviewer’s interests and preferences, landing on a number of papers that stood out in one way or another, or, alternatively, exemplified a particular line of work.

Conclusions: CRI is thriving, not only in the familiar major centers of research, but more widely, throughout the world. This is not to pretend that the distribution is uniform, but to highlight the potential for this domain to play a prominent role in supporting progress in medicine, healthcare, and wellbeing everywhere. We conclude with the observation that CRI and its practitioners would make apt stewards of the new medical knowledge that their methods will bring forward.

Keywords

Clinical data - common data models - biomedical data quality - data analytics - artificial intelligence

• References

• 1 Embi PJ, Payne PRO. Clinical Research Informatics: Challenges, Opportunities and Definition for an Emerging Domain. J Am Med Inform Assoc 2009; 16: 316-27
• 2 Embi P. AMIA CRI Years in Review. Presentations available at http://www.embi.net/cri-years-in-review.html
• 3 van Harmelen F. Protégé 2009. Invited Talk. Presentation available at: https://protege.stanford.edu/conference/2009/slides/FrankvanHarmelen_ProtegeConf09.pdf
• 4 LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015; 521: 436-44
• 5 Rajkomar A, Dean J, Kohane I. Machine Learning in Medicine. N Engl J Med 2019; 380: 1347-58
• 6 Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M. et al Scalable and accurate deep learning with electronic health records. NPJ Digit Medi 2018; 1: 18
• 7 Norgeot B, Glicksberg BS, Trupin L, Lituiev D, Gianfrancesco M, Oskotsky B. et al. Assessment of a Deep Learning Model Based on Electronic Health Record Data to Forecast Clinical Outcomes in Patients With Rheumatoid Arthritis. JAMA Netw Open 2019; 2 (03) e190606
• 8 Rao G, Kirley K, Epner P, Zhang Y, Bauer V, Padman R. et al. Identifying, Analyzing, and Visualizing Diagnostic Paths for Patients with Nonspecific Abdominal Pain. Appl Clin Inform 2018; 9: 905-13
• 9 Seymour CW, Kennedy JN, Wang S, Chang CCH, Elliot CF, Xu Z. et al. Derivation, Validation, and Potential Treatment Implications of Novel Clinical Phenotypes for Sepsis. JAMA 2019; 321 (20) 2003-17
• 10 Liu Y, Chen PHC, Krause J, Peng L. How to Read Articles That Use Machine Learning. Users’ Guides to the Medical Literature. JAMA 2019; 322 (18) 1806-16
• 11 Doshi-Velez F, Perlis RH. Evaluating Machine Learning Articles. JAMA 2019; 322 (18) 1777-9
• 12 Matheny ME, Whicher D, Israni ST. Artificial Intelligence in Health Care- A Report From the National Academy of Medicine. JAMA 2019, Dec 17. Online ahead of print.
• 13 Matheny M, Israni ST, Ahmed M, Whicher D. Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril. NAM Special Publication. Washington, DC: National Academy of Medicine; 2019. Available at: https://nam.edu/artificial-intelligence-special-publication/
• 14 Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data. JAMA Intern Med 2018; 178 (11) 1544-7
• 15 Parikh RB, Teeple S, Navathe AS. Addressing Bias in Artificial Intelligence in Health Care. JAMA 2019; 322 (24) 2377-8
• 16 Nicholson Price W II, Gerke S, Cohen IG. Potential Liability for Physicians Using Artificial Intelligence. JAMA 2019; 322 (18) 1765-6
• 17 Hwang TJ, Kesselheim AS, Vokinger KN. Lifecycle Regulation of Artificial Intelligence– and Machine Learning–Based Software Devices in Medicine. JAMA 2019; 322 (23) 2285-6
• 18 Samek W, Wiegand T, Müller KR. Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models. ITU Journal: ICT Discoveries 2017; Special Issue 1. International Telecommunication Union; 2018.
• 19 Samek W, Montavon G, Vedaldi A, Hansen LK, Klaus-Robert Müller KR. Explainable AI- Interpreting, Explaining and Visualizing Deep Learning LNAI 11700. Springer; 2019
• 20 Adadi A, Berrada M. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access October 2018.
• 21 Tjoa E, Guan C. A Survey on Explainable Artificial Intelligence (XAI): towards Medical XAI. https://arxiv.org/abs/1907.07374
• 22 Insel TR. Digital Phenotyping- Technology for a New Science of Behavior. JAMA 2017; 318 (13) 1215-6
• 23 Miner AS, Milstein A, Hancock JT. Talking to Machines About Personal Mental Health Problems. JAMA 2017; 318 (13) 1217-8
• 24 Nundy S, Montgomery T, Wachter RM. Promoting Trust Between Patients and Physicians in the Era of Artificial Intelligence. JAMA 2019; 322 (06) 497-8
• 25 Verghese A, Shah NH, Harrington RA. What This Computer Needs Is a Physician- Humanism and Artificial Intelligence. JAMA 2018; 319 (01) 19-20
• 26 Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol 2019; 110: 12-22
• 27 Liu VX, Bates DW, Wiens J, Shah NH. The number needed to benefit: estimating the value of predictive analytics in healthcare. J Am Med Inform Assoc 2019; 26 (12) 1655–9
• 28 Van Calster B, Wynants L, Timmerman D, Steyerberg EW, Collins GS. Predictive analytics in health care: how can we know it works?. J Am Med Inform Assoc 2019; 26 (12) 1651-4
• 29 Newcomer SR, Xu S, Kulldorff M, Daley MF, Fireman B, Glanz JM. A primer on quantitative bias analysis with positive predictive values in research using electronic health data. J Am Med Inform Assoc 2019; 26 (12) 1664-74
• 30 Reps JM, Schuemie MJ, Suchard MA, Ryan PB, Rijnbeek PR. Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. J Am Med Inform Assoc 2018; 25 (08) 969-75
• 31 Institute of Medicine (US) Roundtable on Evidence-Based Medicine; Olsen L, Aisner D, McGinnis JM. The Learning Healthcare System: Workshop Summary. Washington (DC): National Academies Press (US); 2007. (This was the first of a number of Workshop Reports issued by the Roundtable on Evidence-Based Medicine. See also 32 below.)
• 32 Institute of Medicine. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary. Washington, DC: The National Academies Press; 2011. https://doi.org/10.17226/12912
• 33 Evans BJ, Krumholz HM. People-powered data collaboratives: fueling data science with the health-related experiences of individuals. J Am Med Inform Assoc 2019; 26 (02) 159-61
• 34 Federal Drug Administration. Real World Evidence. https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence
• 35 Block JP, Bailey LC, Gillman MW, Lunsford D, Boone-Heinonen J, Cleveland LP. et al. PCORnet Antibiotics and Childhood Growth Study: Process for Cohort Creation and Cohort Description. Acad Pediatr 2018; 18 (05) 569-76
• 36 Block JP, Bailey LC, Gillman MW, Lunsford D, Daley MF, Eneli I. et al. Early Antibiotic Exposure and Weight Outcomes in Young Children. Pediatrics 2018; 142 (06) e20180290
• 37 Heerman WJ, Daley MF, Boone-Heinonen J, Rifas-Shiman SL, Bailey CL, Forrest CB. et al. Maternal antibiotic use during pregnancy and childhood obesity at age 5 years. Int J Obes (Lond) 2019; 43: 1202-9
• 38 Lipstein EA, Block JP, Dodds C, Forrest CB, Heerman WJ, Law JK. et al. Early Antibiotics and Childhood Obesity: Do Future Risks Matter to Parents and Physicians?. Clin Pediatr (Phila) 2019; 58 (02) 191-8
• 39 Lin PD, Daley MF, Boone-Heinonen J, Rifas-Shiman SL, Bailey CL, Forrest CB. et al; PCORnet Antibiotics and Childhood Growth Study Group. Comparing Prescribing and Dispensing Data of the PCORnet Common Data Model Within PCORnet Antibiotics and Childhood Growth Study. EGEMS (Wash DC) 2019; 7 (01) 11
• 40 Toh S, Rifas-Shiman SL, Lin PD, Bailey CL, Forrest CB, Horgan CE. et al. Privacy-protecting multivariable-adjusted distributed regression analysis for multi-center pediatric study. Pediatr Res 2020; 87 (06) 1086-92
• 41 Tute E, Steiner J. Modeling of ETL-Processes and Processed Information in Clinical Data Warehousing. Stud Health Technol Inform 2018; 248: 204-11
• 42 Tute E, Wulff A, Marschollek M, Gietzelt M. Clinical Information Model Based Data Quality Checks: Theory and Example. Stud Health Technol Inform 2019; 258: 80-4
• 43 Parciak M, Bauer C, Bender T, Lodahl R, Schreiweis B, Tute E. et al. Provenance Solutions for Medical Research in Heterogeneous IT-Infrastructure: An Implementation Roadmap. Stud Health Technol Inform 2019; 264: 298-302
• 44 Juárez D, Schmidt EE, Stahl-Toyota S, Ückert F, Lablan M. A Generic Method and Implementation to Evaluate and Improve Data Quality in Distributed Research Networks. Methods Inf Med 2019; 58 (2-03): 86-93
• 45 Bai L, Meredith R, Burstein F. A data quality framework, method and tools for managing data quality in a health care setting: an action case study. Journal of Decision Systems 2018; 27 (Suppl 1): 144-54
• 46 Ong T, Pradhananga R, Holve E, Kahn MG. A Framework for Classification of Electronic Health Data Extraction-Transformation-Loading Challenges in Data Network Participation. EGEMS 2017; 5 (01) 10
• 47 https://www.ohdsi.org/
• 48 Kahn MG, Callahan TJ, Barnard J, Bauck AE, Brown J, Davidson BN. et al. A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data. EGEMS (Wash DC) 2016; 4 (01) 1244
• 49 Callahan TJ, Bauck AE, Bertoch D, Brown J, Khare R, Ryan PB. et al. A Comparison of Data Quality Assessment Checks in Six Data Sharing Networks. EGEMS (Wash DC) 2017; 5 (01) 8
• 50 Weiskopf NG, Bakken S, Hripcsak G, Weng C. A Data Quality Assessment Guideline for Electronic Health Record Data Reuse. EGEMS (Wash DC) 2017; 5 (01) 14
• 51 Gold S, Batch A, McClure R, Jiang G, Kharrazi H, Saripalle R. et al. Clinical Concept Value Sets and Interoperability in Health Data Analytics. AMIA Annu Symp Proc 2018; 2018: 480-9
• 52 Rogers JR, Callahan TJ, Kang T, Bauck A, Khare R, Brown JS. et al. A Data Element-Function Conceptual Model for Data Quality Checks. EGEMS (Wash DC) 2019; 7 (01) 17
• 53 Weng C. Clinical data quality: a data life cycle perspective. Biostat Epidemiol 2019; 4 (01) 6-14
• 54 Seneviratne MG, Kahn MG, Hernandez-Boussard T. Merging heterogeneous clinical data to enable knowledge discovery. Pac Symp Biocomput 2019; 24: 439-43
• 55 FHIR & Relational Data Mode (http://community.fhir.org/t/fhir-relational-data-model/427) Response by Lloyd McKenzie to a question on the suitability of FHIR to serve as a relational model
• 56 Dobbins NJ, Spital CH, Black RA, Morrison JM, de Veer B, Zampino E. et al. Leaf: an open-source, model-agnostic, data-driven web application for cohort discovery and translational biomedical research. J Am Med Inform Assoc 2020; 27 (01) 109-18
• 57 Gottesman O, Kuivaniemi H, Tromp G, Faucett WA, Li R, Manolio TA. et al. The Electronic Medical Records and Genomics (eMERGE) Network: past, present, and future. Genet Med 2013; 15 (10) 761-71
• 58 Kirby JC, Speltz P, Rasmussen LV, Basford M, Gottesman O, Peissig PL. et al. PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc 2016; Nov; 23 (06) 1046-52
• 59 Denburg MR, Razzaghi H, Bailey LC, Soranno DE, Pollack AH, Dharnidharka VR. et al. Using Electronic Health Record Data to Rapidly Identify Children with Glomerular Disease for Clinical Research. J Am Soc Nephrol 2019; Dec; 30 (12) 2427-35
• 60 Glenn D, Gibson KL. Finding That Needle in the Haystack: Computable Phenotypes. J Am Soc Nephrol 2019; Dec; 30 (12) 2279-80
• 61 Koola JD, Davis SE, Al-Nimri O, Parr SK, Fabbri D, Malin BA. et al. Development of an automated phenotyping algorithm for hepatorenal syndrome. J Biomed Inform 2018; 80: 87-95
• 62 Phenotype Execution Modeling Architecture (PhEMA) http://informatics.mayo.edu/phema/index.php/Main_Page
• 63 Pacheco JA, Rasmussen LV, Kiefer RC, Campion TR, Speltz P, Carroll RJ. et al. A case study evaluating the portability of an executable computable phenotype algorithm across multiple institutions and electronic health record environments. J Am Med Inform Assoc 2018; 25 (11) 1540-6
• 64 Taylor CO, Lemke KW, Richards TM, Roe KD, He T, Arruda-Olson A. et al. Comorbidity Characterization Among eMERGE Institutions: A Pilot Evaluation with the Johns Hopkins Adjusted Clinical Groups® System. AMIA Jt Summits Transl Sci Proc 2019; 2019: 145-52
• 65 Fawcett N, Young B, Peto L, Quan TP, Gillott R, Wu J. et al. ‘Caveat emptor’: the cautionary tale of endocarditis and the potential pitfalls of clinical coding data-an electronic health records study. BMC Med 2019; 17 (01) 169
• 66 Ando T, Ooba N, Mochizuki M, Koide D, Kimura K, Lee SL. et al. Positive predictive value of ICD-10 codes for acute myocardial infarction in Japan: a validation study at a single center. BMC Health Serv Res 2018; 18 (01) 895
• 67 Singh A, Mora J, Panepinto JA. Identification of patients with hemoglobin SS/S 0 thalassemia disease and pain crises within electronic health records. Blood Adv 2018; 2 (11) 1172-9
• 68 Smoller JW. The use of electronic health records for psychiatric phenotyping and genomics. Am J Med Genet B Neuropsychiatr Genet 2018; 177 (07) 601-12
• 69 Pendergrass SA, Crawford DC. Using Electronic Health Records To Generate Phenotypes For Research. Curr Protoc Hum Genet 2019; 100 (01) e80
• 70 Zhang H, He Z, He X, Guo Y, Nelson DR, Modave F. et al. Computable Eligibility Criteria through Ontology-driven Data Access: A Case Study of Hepatitis C Virus Trials. AMIA Annu Symp Proc 2018; 2018: 1601-10
• 71 Banda JM, Seneviratne M, Hernandez-Boussard T, Shah NH. Advances in Electronic Phenotyping: From Rule-Based Definitions to Machine Learning Models. Annu Rev Biomed Data Sci 2018; 1: 53-68
• 72 Yuan C, Ryan PB, Ta C, Guo Y, Li Z, Hardin J. et al. Criteria2Query: a natural language interface to clinical databases for cohort definition. J Am Med Inform Assoc 2019; 26 (04) 294-305
• 73 Huang GD, Bull J, Johnston McKee K, Mahon E, Harper B, Roberts JN. ; CTTI Recruitment Project Team. Clinical trials recruitment planning: A proposed framework from the Clinical Trials Transformation Initiative. Contemp Clin Trials 2018; 66: 74-9
• 74 Velarde KE, Romesser JM, Johnson MR, Clegg DO, Efimova O, Oostema SJ. et al. An initiative using informatics to facilitate clinical research planning and recruitment in the VA health care system. Contemp Clin Trials Commun 2018; 11: 107-12
• 75 Jain NM, Culley A, Knoop T, Micheel C, Osterman T, Levy M. Conceptual Framework to Support Clinical Trial Optimization and End-to-End Enrollment Workflow. JCO Clin Cancer Inform 2019; 3: 1-10
• 76 Park YR, Yoon YJ, Koo H, Yoo S, Choi CM, Beck SH. , et al. Utilization of a Clinical Trial Management System for the Whole Clinical Trial Process as an Integrated Database: System Development. J Med Internet Res 2018; 20 (04) e103
• 77 Shang N, Liu C, Rasmussen LV, Ta CN, Caroll RJ, Benoit B. et al. Making work visible for electronic phenotype implementation: Lessons learned from the eMERGE network. J Biomed Inform 2019; 99: 103293
• 78 Fiske A, Prainsack B, Buyx A. Data Work: Meaning-Making in the Era of Data-Rich Medicine. J Med Internet Res 2019; 21 (07) e11672
• 79 Haynes K, Selvam N, Cziraky MJ. Bidirectional Data Collaborations in Distributed Research. EGEMS (Wash DC) 2016; 4 (02) 1205
• 80 Kho AN, Cashy JP, Jackson KL, Pah AR, Goel S, Boehnke J. et al. Design and implementation of a privacy preserving electronic health record linkage tool in Chicago. J Am Med Inform Assoc 2015; 22 (05) 1072-80
• 81 Bennett TD, Dean JM, Keenan HT, McGlincy MH, Thomas AM, Cook LJ. Linked Records of Children with Traumatic Brain Injury. Probabilistic Linkage without Use of Protected Health Information. Methods Inf Med 2015; 54 (04) 328-37
• 82 Zimmerman LP, Goel S, Sathar S, Gladfelter CE, Onate A, Kane LL. et al. A Novel Patient Recruitment Strategy: Patient Selection Directly from the Community through Linkage to Clinical Data. Appl Clin Inform 2018; 9 (01) 114-21
• 83 CAPriCORN Chicago Homelessness Project (See https://www.capricorncdrn.org/projects/homelessness-project/)
• 84 Kayaalp M. Patient Privacy in the Era of Big Data. Balkan Med J 2018; 35 (01) 8-17
• 85 Hejblum BP, Weber GM, Liao KP, Palmer NP, Churchill S, Shadick NA. et al Probabilistic record linkage of de-identified research datasets with discrepancies using diagnosis codes. Sci Data 2019; 6: 180298
• 86 Hurley PD, Oliver S, Mehta A. Creating longitudinal datasets and cleaning existing data identifiers in a cystic fibrosis registry using a novel Bayesian probabilistic approach from astronomy. PLoS One 2018; 13 (07) e0199815
• 87 Winter A, Stäubert S, Ammon D, Aiche S, Beyan O, Bischoff V. et al. Smart Medical Information Technology for Healthcare (SMITH). Methods Inf Med 2018; 57 (S 01): e92-e105
• 88 Chen F, Jiang X, Wang S, Schilling LM, Meeker D, Ong T. et al. Perfectly Secure and Efficient Two-Party Electronic-Health-Record Linkage. IEEE Internet Comput 2018; 22 (02) 32-41 doi:10.1109/MIC.2018.112102542
• 89 Laud P, Pankova A. Privacy-preserving record linkage in large databases using secure multiparty computation. BMC Med Genomics 2018; 11 (Suppl 4): 84
• 90 Sohail A, Yousaf MM. A proficient cost reduction framework for de-duplication of records in data integration. BMC Med Inform Decis Mak 2016; 16: 42
• 91 Brown AP, Randall SM, Ferrante AM, Semmens JB, Boyd JH. Estimating parameters for probabilistic linkage of privacy-preserved datasets. BMC Med Res Methodol 2017; 17 (01) 95
• 92 Ranbaduge T, Christen P. A scalable privacy-preserving framework for temporal record linkage. Knowl Inf Syst 2020; 62: 45-78
• 93 Australian Government. Open Data Toolkit: Data Linking (Updated August 2018; accessed December 2019.) https://toolkit.data.gov.au/Data_Linking_Information_Series_Contents_page.html
• 94 Mackey TK, Kuo TT, Gummadi B, Clauson KA, Church G, Grishin D. et al. ‘Fit-for-purpose?’ - Challenges and opportunities for applications of blockchain technology in the future of healthcare. BMC Med 2019; 17 (01) 68
• 95 Hussien HM, Yasin SM, Udzir SNI, Zaidan AA, Zaidan BB. A Systematic Review for Enabling of Develop a Blockchain Technology in Healthcare Application: Taxonomy, Substantially Analysis, Motivations, Challenges, Recommendations and Future Direction. J Med Syst 2019; 43 (10) 320
• 96 Zhang A, Lin X. Towards Secure and Privacy-Preserving Data Sharing in e-Health Systems via Consortium Blockchain. J Med Syst 2018; 42: 140
• 97 Dubovitskaya A, Novotny P, Xu Z, Wang F. Applications of Blockchain Technology for Data-Sharing in Oncology: Results from a Systematic Literature Review. Oncology 2019 Dec 3;1-9
• 98 Hylock RH, Zeng X. A Blockchain Framework for Patient-Centered Health Records and Exchange (HealthChain): Evaluation and Proof-of-Concept Study. J Med Internet Res 2019; 21 (08) e13592
• 99 Duan R, Boland MR, Liu Z, Liu Y, Chang HH, Xu H. et al. Learning from electronic health records across multiple sites: A communication-efficient and privacy-preserving distributed algorithm. J Am Med Inform Assoc 2020; 27 (03) 376-85
• 100 Tong J, Duan R, Li R, Scheuemie MJ, Moore JH, Chen Y. Robust-ODAL: Learning from heterogeneous health systems without sharing patient-level data. Pac Symp Biocomput 2020; 25: 695-706
• 101 Rogers J, Bater J, He X, Machanavajjhala A, Suresh M, Wang X. Privacy Changes Everything . VLDB Poly’19 Towards Polystores that manage multiple Databases, Privacy, Security and/or Policy Issues for Heterogenous Data. VLDB’19, the 45th International Conference on Very Large Data Bases, Los Angeles, California - August 26-30, 2019. Available at: http://users.eecs.northwestern.edu/~jennie/pubs/privacy-changes-everything.pdf
• 102 Copeland R, Mattioli D, Evans M. Paging Dr. Google: How the Tech Giant Is Laying Claim to Health Data. Wall Street Journal, January 11, 2020. https://www.wsj.com/articles/paging-dr-google-how-the-tech-giant-is-laying-claim-to-health-data-11578719700 . (Includes Dr. David Feinberg’s first interview as VP for Google Health.)
• 103 Zuboff S. The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. New York: Public Affairs; 2019.
• 104 Information Commissioner’s Office. Royal Free - Google DeepMind trial failed to comply with data protection law. July 2017. Available at: https://ico.org.uk/about-the-ico/news-and-events/news-and-blogs/2017/07/royal-free-google-deepmind-trial-failed-to-comply-with-data-protection-law/
• 105 Cohen JK. Google, UChicago Med sued over data-sharing practices. Modern Healthcare, June 27, 2019. https://www.modernhealthcare.com/node/952326/printable/print Cohen JK. Google, Ascension data partnership sparks federal probe. Modern Healthcare, November 13, 2019. https://www.modernhealthcare.com/node/959776/printable/print
• 106 Chevrier R, Foufi V, Gaudet-Blavignac C, Robert A, Lovis C. Use and Understanding of Anonymization and De-Identification in the Biomedical Literature: Scoping Review. J Med Internet Res 2019; 21 (05) e13484
• 107 Yoo J, Thaler A, Sweeney L, Zang J. Risks to Patient Privacy: A Re-identification of Patients in Maine and Vermont Statewide Hospital Data. Technology Science. 2018100901. October 09, 2018. https://techscience.org/a/2018100901
• 108 Janmey V, Elkin PL. Re-Identification Risk in HIPAA De-Identified Datasets: The MVA Attack. AMIA Annu Symp Proc 2018; 2018: 1329-37
• 109 Simon GE, Shortreed SM, Coley RY, Penfold RB, Rossom RC, Waitzfelder BE. et al. Assessing and Minimizing Re-identification Risk in Research Data Derived from Health Care Records. EGEMS (Wash DC) 2019; 7 (01) 6
• 110 Na L, Yang C, Lo C, Zhao F, Fukuoka Y, Aswani A. Feasibility of Reidentifying Individuals in Large National Physical Activity Data Sets From Which Protected Health Information Has Been Removed With Use of Machine Learning. JAMA Netw Open 2018; 1 (08) e186040
• 111 McCoy TH, Hughes MC. Preserving Patient Confidentiality as Data Grow: Implications of the Ability to Reidentify Physical Activity Data. JAMA Netw Open 2018; 1 (08) e186029
• 112 Belson NA. FDA’s Historical Use of “Real World Evidence”. Food and Drug Law Institute, Update Magazine, August/September 2018. https://www.fdli.org/2018/08/update-fdas-historical-use-of-real-world-evidence
• 112a See also in the same issue: Onwudiwe NC, Tenenbaum K, Boise BH, Elton J, Manning M. Real World Evidence: Implications and Challenges for Medical Product Communications in an Evolving Regulatory Landscape. Food and Drug Law Institute, Update Magazine, August/September 2018. https://www.fdli.org/2018/08/update-real-world-evidence-implications-and-challenges-for-medical-product-communications-in-an-evolving-regulatory-landscape/
• 113 Law JH, Pettengell C, Le LW, Aviv S, De Marco P, Merritt DC. et al. Generating real-world evidence: Using automated data extraction to replace manual chart review. J Clin Oncol 2019; 37 (15) e18096
• 114 Zhang R, Simon G, Yu F. Advancing Alzheimer’s research: A review of big data promises. Int J Med Inform 2017; 106: 48-56
• 115 He Z, Rizvi RF, Yang F, Adam TJ, Zhang R. Comparing the Study Populations in Dietary Supplement and Drug Clinical Trials for Metabolic Syndrome and Related Disorders. AMIA Jt Summits Transl Sci Proc 2019; 2019: 799-808
• 116 Patorno E, Gopalakrishnan C, Franklin JM, Brodovicz KG, Masso-Gonzalez E, Bartels DB. et al. Claims-based studies of oral glucose-lowering medications can achieve balance in critical clinical variables only observed in electronic health records. Diabetes Obes Metab 2018; 20 (04) 974-84
• 117 Sobel RE, Bate A, Reynold F. Real World Evidence: Time for a Switch?. Drug Saf 2018; 41: 1309-12
• 118 Hripcsak G, Ryan PB, Duke JD, Shah NH, Park RW, Huser V. et al. Characterizing treatment pathways at scale using the OHDSI network. Proc Natl Acad Sci U S A 2016; 113 (27) 7329-36
• 119 Reps JM, Schuemie MJ, Suchard MA, Ryan PB, Rijnbeek PR. Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. J Am Med Inform Assoc 2018; 25 (08) 969-75
• 120 Schuemie MJ, Ryan PB, Hripcsak G, Madigan D, Suchard MA. Improving reproducibility by using high-throughput observational studies with empirical calibration. Philos Trans A Math Phys Eng Sci 2018; 376 2128 20170356
• 121 von Lucadou M, Ganslandt T, Prokosch HU, Toddenroth D. Feasibility analysis of conducting observational studies with the electronic health record. BMC Med Inform Decis Mak 2019; 19 (01) 202
• 122 Hernán MA, Solomonides J, Healy B. A Second Chance to Get Causal Inference Right: A Classification of Data Science Tasks. CHANCE 2019; 32 (01) 42-9
• 123 Hernán MA. The C-Word: Scientific Euphemisms Do Not Improve Causal Inference From Observational Data. Am J Public Health 2018; 108 (05) 616-9
• 123a Begg MD, March D. Cause and Association: Missing the Forest for the Trees. Am J Public Health 2018; 108 (05) 620
• 123b Chiolero A. Data Are Not Enough-Hurray For Causality. Am J Public Health 2018; 108 (05) 622
• 123c Glymour MM, Hamad R. Causal Thinking as a Critical Tool for Eliminating Social Inequalities in Health. Am J Public Health 2018; 108 (05) 623
• 123d Jones HE, Schooling CM. Let’s Require the “T-Word”. Am J Public Health 2018; 108 (05) 624
• 123e Hernán M. The C-Word: The More We Discuss It, the Less Dirty It Sounds. Am J Public Health 2018; 108 (05) 625-6
• 124 Lenert MC, Matheny ME, Walsh CG. Prognostic models will be victims of their own success, unless…. J Am Med Inform Assoc 2019; 26 (12) 1645-50
• 124a Sperrin M, Jenkins D, Martin GP, Peek N. Explicit causal reasoning is needed to prevent prognostic models being victims of their own success. J Am Med Inform Assoc 2019; 26 (12) 1675-6
• 124b Lenert MC, Matheny ME, Walsh CG. Explicit causal reasoning is preferred, but not necessary for pragmatic value. J Am Med Inform Assoc 2019; 26 (12) 1677-8
• 125 Pearl J, Mackenzie D. The Book of Why: The New Science of Cause and Effect. New York: Basic Books; 2018
• 126 Hernán MA, Robins JM. Causal Inference: What If. Boca Raton: Chapman & Hall/CRC. Forthcoming 2020. (A preliminary version is available online at: https://cdn1.sph.harvard.edu/wp-content/uploads/sites/1268/2019/11/ci_hernanrobins_10nov19.pdf ).