Appl Clin Inform 2023; 14(04): 670-683
DOI: 10.1055/a-2105-6505
Special Section on Patient Engagement in Informatics

Engaging Patients and Other Stakeholders in “Designing for Dissemination” of Record Linkage Methods and Tools

Jenna E. Reno
1   RTI International, Center for Communication and Engagement Research, Research Triangle Park, North Carolina, United States
2   School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, United States
Toan C. Ong
2   School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, United States
Chan Voong
2   School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, United States
Brad Morse
2   School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, United States
Kate Ytell
2   School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, United States
Ramona Koren
2   School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, United States
Bethany M. Kwan
2   School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, United States
3   Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, Aurora, Colorado, United States
› Author Affiliations
Funding Research reported in this publication was funded through Patient-Centered Outcomes Research Institute (PCORI) Awards: ME-2018C1-11287. The statements presented in this publication are solely the responsibility of the author(s) and do not necessarily represent the views of the PCORI, its Board of Governors, or Methodology Committee.


Background Novel record linkage (RL) methods have the potential to enhance clinical informatics by integrating patient data from multiple sources—including electronic health records, insurance claims, and digital health devices—to inform patient-centered care. Engaging patients and other stakeholders in the use of RL methods in patient-centered outcomes research (PCOR) is a key step in ensuring RL methods are viewed as acceptable, appropriate, and useful. The University of Colorado Record Linkage (CURL) platform empowers the use of RL in PCOR.

Objectives This study aimed to describe the process of engaging patients and other stakeholders in the design of an RL dissemination package to support the use of RL methods in PCOR.

Methods Customer discovery, value proposition design, and user experience methods were used to iteratively develop an RL dissemination package that includes animated explainer videos for patients and an RL research planning workbook for researchers. Patients and other stakeholders (researchers, data managers, and regulatory officials) were engaged in the RL dissemination package design.

Results Patient partners emphasized the importance of conveying how RL methods may benefit patients and the rules researchers must follow to protect the privacy and security of patient data. Other stakeholders described accuracy, flexibility, efficiency, and data security compared with other available RL solutions. Dissemination package communication products reflect the value propositions identified by key stakeholders. As prioritized by patients, the animated explainer videos emphasize the data privacy and security processes and procedures employed when performing research using RL. The RL workbook addresses researchers' and data managers' needs to iteratively design RL projects and provides accompanying resources to alleviate leadership and regulatory officials' concerns about data regulation compliance.

Conclusion Dissemination products to promote adoption and use of CURL include materials to facilitate patient engagement in RL research and investigator step-by-step decision-making materials about the integration of RL methods in PCOR.

Protection of Human and Animal Subjects

The study was performed in compliance with the U.S. Title 45 Code of Federal Regulations Part 46 and was reviewed by Colorado Multiple Institutional Review Board.

Authors' Contributions

J.E.R. led the drafting of this manuscript as well as conducting or otherwise supporting all stakeholder engagement and design methods and analysis. T.C.O. was the principal investigator on the funding award that supported this work; he provided technical expertise on the integration of all RL content in the dissemination package as well as coplanning and facilitating stakeholder engagement efforts including the RL workshop and town hall. C.V. provided project management for stakeholder engagement activities. B.M. led the planning, implementation, and analysis of user-centered design think aloud usability interviews, contributed to the finalization of the CURL Workbook, and participated in the drafting of this manuscript. K.Y. supported the qualitative analysis of customer discovery interviews as well as cofacilitated think aloud usability interviews and participated in the analysis and integrations insights from these sessions. R.K. served as the patient partner on the design team and contributed to all stakeholder engagement and design efforts. B.K. directed all stakeholder engagement activities and the integration of findings into the design process, as well as writing on this manuscript.

Publication History

Received: 23 September 2022

Accepted: 01 June 2023

Accepted Manuscript online:
05 June 2023

Article published online:
23 August 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Meystre SM, Lovis C, Bürkle T, Tognola G, Budrionis A, Lehmann CU. Clinical data reuse or secondary use: current status and potential future progress. Yearb Med Inform 2017; 26 (01) 38-52
  • 2 Ong TC, Kahn MG, Kwan BM. et al. Dynamic-ETL: a hybrid approach for health data extraction, transformation and loading. BMC Med Inform Decis Mak 2017; 17 (01) 134
  • 3 Jiang G, Kiefer RC, Sharma DK, Prud'hommeaux E, Solbrig HR. A consensus-based approach for harmonizing the OHDSI common data model with HL7 FHIR. Stud Health Technol Inform 2017; 245: 887-891
  • 4 Kahn MG, Callahan TJ, Barnard J. et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC) 2016; 4 (01) 1244
  • 5 Richter A, Schmidt CO, Krüger M, Struckmann S. DataquieR: assessment of data quality in epidemiological research. J Open Source Softw 2021; 6 (61) 3093
  • 6 Boyd JH, Randall SM, Ferrante AM. Application of privacy-preserving techniques in operational record linkage centres. Med Data Privacy Handbook Switzerland: Springer International Publishing; 2015
  • 7 Brown AP, Ferrante AM, Randall SM, Boyd JH, Semmens JB. Ensuring privacy when integrating patient-based datasets: new methods and developments in record linkage. Front Public Health 2017; 5: 34
  • 8 Kohane IS, Aronow BJ, Avillach P. et al; Consortium For Clinical Characterization Of COVID-19 By EHR (4CE). What every reader should know about studies using electronic health record data but may be afraid to ask. J Med Internet Res 2021; 23 (03) e22219
  • 9 Ong TC, Duca LM, Kahn MG, Crume TL. A hybrid approach to record linkage using a combination of deterministic and probabilistic methodology. J Am Med Inform Assoc 2020; 27 (04) 505-513
  • 10 Lim D, Randall S, Robinson S. et al. Unlocking potential within health systems using privacy-preserving record linkage: exploring chronic kidney disease outcomes through linked data modelling. Appl Clin Inform 2022; 13 (04) 901-909
  • 11 Ross MK, Sanz J, Tep B, Follett R, Soohoo SL, Bell DS. Accuracy of an electronic health record patient linkage module evaluated between neighboring academic health care centers. Appl Clin Inform 2020; 11 (05) 725-732
  • 12 Mason LG. A Comparison of Record Linkage Techniques. Statistical Survey Paper. Quarterly Census of Wages and Employment (QCEW). 2018 Accessed August 31, 2022 at:
  • 13 Hall R, Fienberg SE. Privacy-preserving record linkage. In: Privacy in Statistical Databases. Berlin, Heidelberg: Springer; 2010: 269-283
  • 14 Conway RBN, Armistead MG, Denney MJ, Smith GS. Validating the matching of patients in the linkage of a large hospital system's EHR with state and national death databases. Appl Clin Inform 2021; 12 (01) 82-89
  • 15 Kwan BM, Brownson RC, Glasgow RE, Morrato EH, Luke DA. Designing for dissemination and sustainability to promote equitable impacts on health. Annu Rev Public Health 2022; 43 (43) 331-353
  • 16 Hartzler A, McCarty CA, Rasmussen LV. et al. Stakeholder engagement: a key component of integrating genomic information into electronic health records. Genet Med 2013; 15 (10) 792-801
  • 17 Baumann BL, Kolko DJ, Collins K, Herschell AD. Understanding practitioners' characteristics and perspectives prior to the dissemination of an evidence-based intervention. Child Abuse Negl 2006; 30 (07) 771-787
  • 18 Zimmerman LP, Goel S, Sathar S. et al. A novel patient recruitment strategy: patient selection directly from the community through linkage to clinical data. Appl Clin Inform 2018; 9 (01) 114-121
  • 19 Morrato EH, Hamer MK, Sills M, Kwan B, Schilling LM. Applying a commercialization-readiness framework to optimize value for achieving sustainability of an electronic health data research network and its data capabilities: the SAFTINet experience. EGEMS (Wash DC) 2019; 7 (01) 48
  • 20 Concannon TW, Meissner P, Grunbaum JA. et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med 2012; 27 (08) 985-991
  • 21 Lopez MH, Holve E, Rein A, Winkler J. Involving patients and consumers in research: new opportunities for meaningful engagement in research and quality improvement. In: EDM Forum,. Washington, DC: AcademyHealth; 2012
  • 22 Lavallee DC, Wicks P, Alfonso Cristancho R, Mullins CD. Stakeholder engagement in patient-centered outcomes research: high-touch or high-tech?. Expert Rev Pharmacoecon Outcomes Res 2014; 14 (03) 335-344
  • 23 McGuire WJ. McGuire's Classic Input–Output Framework for Constructing Persuasive Messages. Public Communication Campaigns. 2013. Thousand Oaks, CA: SAGE Publications; 133-145
  • 24 Ong TC, Hill AC, Grannis S, Voong C, Kahn MG. Linkability assessment for demographics data. In: AMIA 2020 Virtual Informatics Summit Proceedings. American Medical Informatics Society; 2020
  • 25 Ong TC, Hill AC, Kahn M, Grannis S, Voong C, Shilling L. Privacy-Preserving Transitive Record Linkage. In: AMIA 2021 Virtual Informatics Summit Proceedings. American Medical Informatics Society; 2021
  • 26 CU record linkage. University of Colorado record linkage. DOI: 10.25677/10968/6173
  • 27 Osterwalder A, Pigneur Y, Bernarda G, Smith A. Value Proposition Design: How to Create Products and Services Customers Want. John Wiley & Sons; 2015
  • 28 Gray CM. “It's More of a Mindset Than a Method”: UX Practitioners' Conception of Design Methods. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2016: 4044-4055
  • 29 Hartson R, Pyla PS. The UX Book. Agile UX Design for a Quality User Experience. Morgan Kaufmann 2018
  • 30 / What is plain language?. Accessed January 1, 2023 at:
  • 31 Kirwan JR, de Wit M, Frank L. et al. Emerging guidelines for patient engagement in research. Value Health 2017; 20 (03) 481-486
  • 32 CU Record Linkage. iPPRL value proposition design customer discovery phone interviews: investigators, researchers, and clinical operations. CU Anschutz Digital Collections 2021 DOI: 10.25677/10968/6180
  • 33 CU Record Linkage. iPPRL value proposition design customer discovery phone interviews: data manager/technician. CU Anschutz Digital Collections 2021 DOI: 10.25677/10968/6178
  • 34 CU Record Linkage. iPPRL value proposition design customer discovery phone interviews: leadership and regulatory. CU Anschutz Digital Collections 2021 DOI: 10.25677/10968/6182
  • 35 Scott KW, Howell D. Clarifying analysis and interpretation in grounded theory: using a conditional relationship guide and reflective coding matrix. Int J Qual Methods 2008; 7 (02) 1-15
  • 36 Kepper MM, Walsh-Bailey C, Brownson RC. et al. Development of a health information technology tool for behavior change to address obesity and prevent chronic disease among adolescents: designing for dissemination and sustainment using the ORBIT model. Front Digit Health 2021; 3: 648777
  • 37 Haddad R, Badke D'Andrea C, Ricchio A. et al. Using Innovation-Corps (I-Corps™) Methods to Adapt a Mobile Health (mHealth) Obesity Treatment for Community Mental Health Settings. Front Digit Health 2022; 4: 835002
  • 38 CU record linkage: patient data and privacy regulations. CU Anschutz Digital Collections 2021 DOI: 10.25677/10968/6184
  • 39 CU record linkage: privacy-protected record linkage. CU Anschutz Digital Collections 2021 DOI: 10.25677/10968/6174
  • 40 CU Record Linkage. Record linkage for investigators. CU Anschutz Digital Collections 2021 DOI: 10.25677/10968/6183
  • 41 CU Record Linkage. Health data linkage workshop materials. CU Anschutz Digital Collections 2021 DOI: 10.25677/10968/6176
  • 42 Lewis C. Using the” thinking-aloud” method in cognitive interface design. Yorktown Heights, NY: IBM T.J. Watson Research Center; 1982
  • 43 Meissner P, Cottler LB, Eder MM, Michener JL. Engagement science: the core of dissemination, implementation, and translational research science. J Clin Transl Sci 2020; 4 (03) 216-218