Characterization of Safety Events Involving Technology in Primary and Community Care

Abstract

Background The adoption of technology in health care settings is often touted as an opportunity to improve patient safety. While some adverse events can be reduced by health information technologies, technology has also been implicated in or attributed to safety events. To date, most studies on this topic have focused on acute care settings.

Objectives To describe voluntarily reported safety events that involved health information technology in community and primary care settings in a large Canadian health care organization.

Methods Two years of safety events involving health information technology (2016–2018) were extracted from an online voluntary safety event reporting system. Events from primary and community care settings were categorized according to clinical setting, type of event, and level of harm. The Sittig and Singh sociotechnical system model was then used to identify the most prominent sociotechnical dimensions of each event.

Results Of 104 reported events, most (n = 85, 82%) indicated the event resulted in no harm. Public health had the highest number of reports (n = 45, 43%), whereas home health had the fewest (n = 7, 7%). Of the 182 sociotechnical concepts identified, many events (n = 61, 59%) mapped to more than one dimension. Personnel (n = 48, 46%), Workflow and Communication (n = 37, 36%), and Content (n = 30, 29%) were the most common. Personnel and Content together was the most common combination of dimensions.

Conclusion Most reported events featured both technical and social dimensions, suggesting that the nature of these events is multifaceted. Leveraging existing safety event reporting systems to screen for safety events involving health information technology, and applying a sociotechnical analytic framework can aid health organizations in identifying, responding to, and learning from reported events.

Keywords

sociotechnical aspects - home health - ambulatory care - primary care - Patient safety

Background and Significance

The integration of technology in health care is often seen as a means to improve patient safety.[1] While technology has been shown to reduce many types of medical errors (e.g., barcode medication administration tools can reduce medication errors),[2] [3] there is a growing body of evidence indicating that technology can also be associated with safety events.[4] [5] [6] [7] [8] These safety events have been studied and labeled with terms such as unintended consequences,[9] e-iatrogenesis,[10] technology-induced error.[7]

Research on the negative unintended consequences of technology in health care has largely been based on acute care settings with minimal research in primary and community care settings.[11] [12] [13] Yet, there is emerging evidence that safety events involving health information technology (HIT) are indeed occurring in primary and community care settings.[6] [14] [15] Errors that may contribute to safety events involving HIT in nonacute settings may stem from various sources such as missing or incorrect data,[16] [17] unreliable software,[6] [18] and poor usability.[19] [20] As health care organizations continue to expand their services and incorporate new technologies in nonacute settings,[21] it is important to understand the potential safety implications of HIT.

Voluntarily reported safety events can offer valuable insights into emerging risks associated with HIT in health care settings.[22] [23] [24] [25] [26] Safety event reporting data have proven to be instrumental in supporting learning and improvement in health care organizations.[1] [27] [28] For safety events involving HIT, recognizing and reporting these situations can help identify opportunities to promote HIT safety and enhance care.[7] [29] For the purpose of this paper, the terms HIT-related safety events and safety events involving HIT are used interchangeably to refer to instances where HIT was involved in safety events, adverse events, or incident reports, as indicated by the person initiating the report. Many health care organizations worldwide have established reporting infrastructures to identify, evaluate, analyze, and improve patient safety.[30] [31] However, HIT-related safety events are still not widely reported and analyzed in health care.[32] [33] Developing real-time methods for HIT safety surveillance remains an ongoing challenge due to the complexity inherent in the design and use of HIT.[4] Applying a sociotechnical framework is a useful approach to conceptualize the complexities of a technologically enabled system and gain insight into the myriad challenges that may be captured in reported safety events.[34]

Objectives

The aim of this study was to investigate the characteristics of voluntarily reported HIT-related safety events in primary and community care settings. While it is acknowledged that voluntary reporting systems may not capture all safety events,[35] [36] analyzing the patterns and characteristics of voluntarily reported safety events can illuminate valuable insights into the safety implications of HIT. The study also involved the participation of health care organization leaders, and the findings were shared with the organization to inform ongoing operations.[37]

Methods

Study Design and Setting

This descriptive study examined 2 years of voluntarily reported safety events from primary and community care settings that indicated computer involvement. The study was conducted in a large health care organization in Western Canada that provides health care services for a population of 1.25 million people, including both densely populated urban areas, as well as smaller communities. The study focused on the urban core of approximately 630,000 people, and the services delivered in nonacute care settings (i.e., home care, primary care clinics, population and public health, residential care, mental health, and substance use services). The primary and community care settings used in this study were defined based on the health care organization's service delivery model. Residential care was excluded from the study due to the similarity in the service delivery model with acute care settings, where patients receive continuous care instead of episodic care. Ethics approval was obtained from the University of British Columbia Behavioral Research Ethics Board.

The organization used a web-based voluntary reporting system for adverse events, which had been in place for more than 10 years.[38] Staff members, primarily but not limited to clinicians and managers, are encouraged to report instances of patient harm or circumstances with the potential to cause harm. The reporting system prompts the reporting staff member to input the details of the situation, including the type of incident and degree of harm that resulted from the event based on the World Health Organization's International Classification for Patient Safety,[39] as well as free-text descriptions of what happened. To capture HIT involvement in reported safety events, the regional online voluntary reporting system was updated in early 2016 to include the question, “Was a computer system involved in this event?”[40]

Data Collection

Data collected for this study included all safety events from primary and community care settings between November 1, 2016 and October 31, 2018 in which the reporting staff member indicated computer involvement. Each report contained: deidentified unique ID, date, location, type of care setting in which the incident occurred, type of incident,[39] degree of harm,[39] and a free-text description of the incident by the reporting staff member including how the computer system contributed to the event. An example of a reported HIT-related safety event is an incident where the electronic health record (EHR) allergy alert system did not function properly, resulting in a patient being administered a medication that was a known allergen and the patient experienced an allergic reaction requiring medical attention. This example report was identified for data extraction because the reporting staff member had selected “yes” to the “computer involvement” screening question.

Data Abstraction and Descriptive Analysis

The data abstraction and analysis were conducted in a systematic fashion to promote accuracy and reliability of the findings. A database custodian from the health care organization manually extracted and deidentified the data, and then shared it with the research team. Categorical data were analyzed using descriptive statistics to summarize the frequencies and proportions of safety events, based on the information available in the report.

Sociotechnical Model Analysis

A content analysis of the free-text descriptions of the safety events and how the computer was involved was conducted using the Sittig and Singh sociotechnical model, which identifies eight technical and nontechnical dimensions related to HIT and the interactions of these dimensions within the context of a complex adaptive system.[5] [41] The eight dimensions include: Hardware and Software; Clinical Content; User Interface; Personnel; Workflow and Communication; Internal Organizational Policies; External Rules and Regulations; and Measurement and Monitoring.[5] [41]

To ensure interrater reliability, an academic researcher (C.R.) and an organizational executive leader from the partnering health organization (M.S.), conducted the sociotechnical model analysis in an iterative process. They first familiarized themselves with the sociotechnical model by reviewing relevant literature,[5] [41] [42] then reviewed a training set of 10 reported safety events together to establish consensus on the most applicable sociotechnical dimensions and the rationale for assigning the codes. The researchers then coded the entire corpus of study data independently in cycles, in batches of 20 events, with checkpoints after each batch to assess agreement levels and resolve discrepancies through discussion.

Interrater Reliability for Sociotechnical Analysis

To assess interrater reliability, both interrater agreement (percent agreement) and Cohen's kappa (κ) were calculated after each cycle of independent coding by the two researchers, and then discrepancies were discussed to reach consensus. Because each event could theoretically have up to eight codes assigned, the percent agreement of the number of individual codes assigned to each event was also calculated. Interrater agreement was calculated according to the formula:

[Eq. 1:] Formula for interrater agreement

Cohen's kappa (κ) and percent agreement are both reported because either one on its own has limitations (i.e., there are discrepancies on acceptable kappa levels and percent agreement does not account for chance agreement).[43] The agreement statistics for each round of coding are presented in [Table 1]. Although the agreement statistics for each round of coding showed very low interrater agreement, it facilitated a robust discussion of the rationale for the codes, and consensus was reached quickly. The goal was ultimately to reach consensus on the sociotechnical dimensions for each event to learn more about what may have led to the safety event and map patterns among the reported events.

Table 1 Interrater agreement on sociotechnical dimensions

Coding round	n	Interrater agreement	Interrater agreement by item	Cohen's kappa ҡ	p-Value	Consensus after discussion
Training	10	–	–	–	–	100%
1	18	33%	58%	0.285	<0.001	100%
2	21	58%	66%	0.578	<0.001	100%
3	19	42%	55%	0.361	<0.001	100%
4	46	50%	66%	0.362	<0.001	100%
Overall	104	45%	57%	0.405	<0.001	100%

Notes: Training events not included in dataset. Interrater agreement calculated based on whether raters' entire coding for the event matched and interrater agreement by item is calculated based on the number of matching individual codes assigned.

Results

Descriptive Analysis of Safety Events Based on World Health Organization Criteria

A total of 104 safety event reports were included in the analysis. Across the different types of settings, safety events from public health had the highest number of reports (n = 45), whereas home health had the fewest (n = 7). The most common type of safety event reported was medication/intravenous fluid/biological (n = 68). The types of safety events across the different care settings can be found in [Table 2]. Of the 104 reports analyzed, 85 (82%) indicated that the safety event resulted in no harm, 13 (13%) resulted in minor harm, 5 (5%) resulted in moderate harm, and 1 (1%) report indicated severe harm. There were no reports of death. The levels of harm reported across the five primary and community care settings is shown in [Table 2].

Table 2 Number of safety events by setting, degree of harm, and type of event

	Home health	Primary care	Mental health	Pharmacy	Public health	Total
Degree of harm
Death	0	0	0	0	0	0
Severe harm	0	0	1	0	0	1
Moderate harm	2	2	1	0	0	5
Minor harm	2	4	2	2	3	13
No harm	3	7	14	19	42	85
Total	7	13	18	21	45	104
Type of event
Unsafe behavior	0	0	1	0	0	1
Medication/intravenous fluid/biological	1	4	13	21	29	68
Laboratory	0	4	0	0	2	6
Equipment/product/medical device	0	0	1	0	0	1
Documentation	0	0	0	0	8	8
Clinical process/procedure	4	3	2	0	4	13
Clinical administration	2	2	1	0	2	7
Total	7	13	18	21	45	104

The safety events identified as “No Harm” were most often related to immunizations, including vaccines administered off schedule, without proper consent, to the wrong person, past expiration date, and/or were not properly documented. An example of an event reported with minor harm was when a home care client did not receive daily wound care over a weekend, attributed to the fact that recurring appointments could not be entered into the EHR schedule. An example of a safety event that was reported as moderate harm was an incorrect referral to a social worker rather than a registered nurse that led to incomplete contact tracing and follow-up care concerning a communicable disease.

Finally, the single safety event where severe harm was reported involved a client who had been redirected from a community clinical site to seek immediate attention at an emergency department after critical laboratory values were noted in their EHR. The laboratory values were not relayed to, nor accessed by, the emergency department and the client was immediately released without treatment. There were no details regarding the client's outcome in the report and it was unclear specifically why the reporting clinician indicated severe harm. However, the report indicated that the computer system contributed to the safety event because the client's EHR data from the community setting containing the critical laboratory values were not readily accessible and not viewed by the emergency department staff.

Analysis of Safety Events Using Sociotechnical Analysis

A total of 182 sociotechnical dimensions were identified within the 104 safety events. The most commonly assigned dimension was Personnel (n = 48), followed by Workflow and Communication (n = 37), and then Content (n = 30). The total number of times each sociotechnical dimension code was assigned for the entire dataset is presented in [Fig. 1], and examples to illustrate reports featuring each of the eight sociotechnical dimensions are shown in [Table 3]. Of the 182 sociotechnical dimensions identified, many events (n = 61, 59%) mapped to more than one dimension. The most commonly assigned code was Personnel, followed by Personnel and Clinical Content together. The most common combinations of sociotechnical dimensions that were coded are shown in [Fig. 2].

Examples from the dataset to illustrate the assignment of sociotechnical dimensions, and the full list of dimensions assigned are available in the supplementary materials. ([Supplementary S1] and [S2], available in the online version only)

Fig. 1 Sociotechnical dimensions: individual.Note: 104 events analyzed, with 1 to 4 dimensions identified for each event, for a total of 182 dimensions identified.

Fig. 2 Most common sociotechnical dimensions: individual and Combinations. Note: only categories with 5 or more events shown.

Table 3 Examples of safety events by sociotechnical dimension featured

	Dimension	Description (adapted from Singh and Sittig, 2020; Sittig and Singh, 2010, 2017.)	Example safety event
1	Hardware and software	Purely technical, the physical devices, and software	Client database was slow to load, leading to a client who had been barred from accessing clinic being granted access because the clinician could not view the client's history in the database
2	Clinical content	All textual data stored within the system	Limited options for referral reason in EHR, leading to missed care because assigned care provider was anticipating wound care and not qualified to administer chemotherapy as client required
3	User interface	Aspects of the computer that the user can touch, see, and hear	Alert fatigue, leading to an alert to follow-up with police remaining active for 16 encounters before it was addressed
4	Personnel	Humans involved in the design, development, implementation, and use of HIT, purely social	User documented on the wrong client's chart, leading to a client receiving a prescription with another client's name, birthdate, etc.
5	Workflow and communication	Processes to deliver care effectively, people need to work together	Two different clinical services using the same HIT system with ambiguity over who was responsible for certain laboratory results, leading to the most responsible care provider remaining unaware of significant results
6	Internal organizational policies, procedures, and culture	Affects all dimensions in the model, influence of leadership and resources. Aligns with external regulation	Client information was not successfully transferred from one service to another, leading to missed care such as central line care or breastfeeding support
7	External rules and regulations	Forces that facilitate or constrain HIT in the clinical setting	Care provider did not have access to pertinent client information from provincial medication database, leading to a client deemed high risk for overdose having to go without opioid agonist therapy for several days
8	Measurement and monitoring	Assessing the effects of HIT on an ongoing basis	HIT system downtime prevented trigger to resume community services following a client's discharge from acute care, leading to decline in their clinical status

Abbreviations: EHR, electronic health record; HIT, health information technology.

Discussion

This study used descriptive analysis and sociotechnical analysis to characterize computer-related safety events reported from primary and community care settings via a voluntary safety event reporting system. Two years of data from primary and community care settings were analyzed and it was found that most events resulted in no harm. Reports were most commonly generated from public health settings, and medication-related events were the most common type of adverse event reported. The findings of this study provide an important contribution to advancing knowledge about the characteristics of HIT-related safety events within the contexts of primary and community care.

Sociotechnical Analysis

The analysis of 104 reported events identified a total of 182 dimensions matched to the sociotechnical model, and the specific dimensions of Personnel and Workflow and Communication were most commonly recognized. Most events featured both technical and social dimensions, suggesting that the nature of these events is multifaceted, and that the technology, the users, and the context, and interactions among these components, are all important to consider in analyzing and addressing safety events involving HIT. The sociotechnical model was a valuable and pragmatic tool to optimize the organizational learning from safety events involving HIT. The model helped ensure safety events were reviewed systematically to identify all social and technical aspects related to each event and extract information about the contributing factors that might otherwise remain latent.

Only a small number of previous studies have applied the Sittig and Singh model to analyze HIT-related safety issues, and these studies have divergent findings. For primary and community care settings specifically, a recent study employed the Sittig and Singh sociotechnical model to classify safety events related to HIT. Powell et al[14] analyzed 214 reports of root cause analyses of adverse events related to HIT in an outpatient setting to examine the HIT-related factors associated with diagnostic delay. They determined that the predominant sociotechnical dimensions were Personnel, and Workflow and Communication. This is similar to studies focused on acute care settings[44] [45] and is also consistent with the findings of the present study. The similarity of these findings suggests the most significant risks that HIT inadvertently may pose to patient safety is related to how the technology fits the needs of the users and the context of use and not merely the technology itself.

A study from 2014 by Meeks et al[44] analyzed 100 internal investigation reports from a large integrated health system that included acute care, primary care, and community care and reported that the most commonly associated dimensions were Hardware and Software, Clinical Content, and Workflow and Communication.[44] Data from the Meeks et al study were not disaggregated by setting, so the impact specifically on primary and community care is not clear.

A second study from 2016 by Castro et al[45] examined 120 HIT-related sentinel events reported to the Joint Commission from accredited organizations in the United States.[45] This study found User Interface, Workflow and Communication, and Clinical Content were the most commonly identified sociotechnical dimensions.[45] The top three most commonly identified dimensions in the Meeks et al and Castro et al studies included both technical and social dimensions, which contrasts with a 2018 study by Kang et al[46] that found the most commonly identified dimensions were technical dimensions. In that recent study, Kang et al[46] applied the sociotechnical model to analyze 268 HIT-related safety events reported to the Federal Drug Administration Manufacturer and User Facility Device Experience (MAUDE) database and found that these safety events were most commonly related to the dimensions of Hardware and Software, Clinical Content, and User Interface. The variation between these studies may reflect differences in the databases from which the data were extracted. The MAUDE database is intended for reports related to devices and therefore may contain more information related to technical dimensions, whereas safety event reporting and incident report investigations may collect more contextual and clinical details. This highlights that differences in the way information is collected and the perspectives of the people reporting the events may influence what insights are gleaned from reported events. Finally, in a retrospective analysis of safety huddle discussions to identify safety risks of EHRs, Menon et al[47] applied the sociotechnical model to 245 identified concerns from a hospital setting and found that three dimensions, Hardware and Software, Clinical Content, and Personnel, accounted for nearly three quarters of the safety issues identified.

Interestingly, none of the studies that used the Sittig and Singh model reported the combinations of dimensions for items that had more than one dimension assigned. Yet, in the current study, Personnel had the largest proportion of individual events, followed by Personnel combined with Clinical Content, highlighting the interrelatedness of the dimensions of the model.[5] [41] Recognizing the frequency of a combination of dimensions assigned in the dataset helps to reinforce the context of a complex system and supports a multipronged approach to remediating and preventing safety events. For example, for events indicating dimensions of Personnel and Clinical Content, following up with additional training for users may be helpful, but a comprehensive response would also include addressing issues related to clinical content. Future activities that use this model should explore the interrelatedness of these dimensions, as there may be relevant patterns that could be more informative than the individual dimensions alone.

Frequencies of Events

The frequencies of events and incidents of harm in this study are lower than most estimates of adverse events and harm in the literature.[48] [49] [50] There have been fewer investigations focused specifically on clinical settings outside hospitals, and the assessed frequencies of instances of harm in nonacute settings varies greatly.[51] [52] [53] While the level of risk for harm related to technology may potentially be lower in ambulatory settings given the less acute nature of patient illness in ambulatory care,[15] more research is warranted to examine the prevalence and consequences of HIT-related safety events in nonacute settings. Further, given the more episodic nature of care and greater autonomy of patients, the instances of harm may be harder to capture. Less overt instances of harm such as a delayed follow-up to an abnormal laboratory result could in fact result in a missed or delayed diagnosis, failure or absence of treatment, and ultimately, a severe degree of harm.[14] [54] Further to this, future work in this area needs to acknowledge the patient's agency and perceptions of safety, including the social and psychological impacts of care, in addition to physical harm, in recognizing and addressing safety concerns.[55] Future research exploring HIT safety from the patient perspective may also provide valuable insights in this topic area.

Voluntary Reporting Systems

Limitations of voluntary safety reporting systems such as underreporting, subjectivity, misinterpretation of reports, and a lack of impact or follow-up have been well documented in the literature.[27] [28] [35] [56] These limitations are echoed in relation to voluntary reporting of HIT-related safety events as well. The details of reported events are subjective, and clinicians may not recognize or be able to fully articulate the role of technology in the safety event; therefore, safety events involving HIT may be underreported.[7] [57] Despite these challenges, this study demonstrates that there is some potential benefit from including a screening question for technology/computer safety in voluntary reporting systems.

Technologies and their applications in care delivery will continue to evolve, and it is critical that health care organizations continue to monitor for unintended consequences and mitigate threats to patient safety. Modifying the reporting system to include a direct screening question about technology can help to glean more information about HIT-related safety events that have occurred and help prioritize improvements to HIT. Future work should consider the exact manner in which HIT-related questioning is constructed. For example, in our system the question directly asked “was a computer involved?” may have limited responses by guiding reporters to only think of computers. It is possible that if the question had asked “was technology involved in this event?,” the safety event reporters may have thought more broadly about all of the technologies in use in their clinical context. Additionally, while the general design of safety event reporting systems is known to have limitations,[58] there is value in focusing on analytic processes and taxonomies that are specific to HIT safety concerns to gain greater insight into the complexity surrounding these issues and guide improvement.[59] [60] Furthermore, knowledge gleaned from reported HIT safety events can inform future efforts to apply artificial intelligence to automate and expediate the process of identifying and responding to HIT-related risks.[61] [62]

Limitations

There are several limitations to this study. First, voluntarily reported safety data have limitations, including incompleteness, subjectivity, and underreporting.[35] [36] Indeed, the findings here are not a comprehensive representation of all safety concerns in practice. Additionally, given this was a secondary use of data, there were some challenges because the data were originally collected for a different purpose.[63] In this study, the analysis may have been limited because the reported safety events involved a variety of different HIT systems, and the specific HIT system involved was not necessarily specified, and/or details related to how HIT played a role in the event were limited. Further to this, the assessment of the degree of harm that resulted in each safety event was limited to that which was initially assigned by the reporting clinician and was difficult to verify based on the limited information available in the reports. Moreover, the initial interrater agreement in determining the related sociotechnical dimensions was low, also potentially related to the limited information available in reports. However, through the process of consensus, all disagreements were easily reconciled. A larger volume of reports may have helped to provide more precision to the coding process. Future research should continue to explore reliability of this coding scheme. Additionally, because it is likely that not all events that occurred were reported,[35] it must be noted that the findings from this study do not represent the actual incidence of HIT-related safety events in primary and community care settings. Finally, the sample size of this study was small and therefore may not have been representative of all types of safety events involving HIT, nor necessarily generalizable to other settings.

Conclusion

The body of literature related to safety events involving HIT is relatively young yet is developing rapidly. To our knowledge, this is the first study to examine data from a voluntary reporting system with a “computer involvement” screening question and the first to examine the intersections of the Sittig and Singh sociotechnical dimensions. There is a need for researchers to continue to develop the body of knowledge related to technology and patient safety.[4] [7] There are calls for further studies to measure the frequency and magnitude of negative effects of technology[64] as well as to take up more innovative approaches to better understand the complex interactions among the sociotechnical elements of the health care system.[4] The Sittig and Singh model can provide a valuable and pragmatic framework for analyzing HIT issues and gleaning insights to support a systems thinking approach.[65]

Clinical Relevance Statement

Clinicians work in close proximity with HIT in delivering care and can identify and report HIT-related safety concerns to help mitigate against harm to patients.

Multiple-Choice Questions

1. What is a HIT-related safety event?

   • A safety event, adverse event, or incident involving HIT

   • An application of technology to improve safety

   • An event to discuss safety and health information technologies

   • None of the above

   Correct Answer: The correct answer is option a. In this paper, the terms HIT-related safety events (used interchangeably with safety events involving HIT), refers to safety events, adverse events, or incident reports where HIT was involved, as indicated by the person initiating the report.

2. Which of the following is NOT a dimension of the Sittig and Singh sociotechnical model?

   • Personnel

   • Measurement and monitoring

   • Hardware and software

   • User preferences

   Correct Answer: The correct answer is option d. The Sittig and Singh sociotechnical model includes eight technical and nontechnical dimensions related to HIT and focuses on their interactions within the context of a complex adaptive system. The eight dimensions are: Hardware and Software; Clinical Content; User Interface; Personnel; Workflow and Communication; Internal Organizational Policies; External Rules and Regulations; and Measurement and Monitoring.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

This research did not involve human subjects.

• References

• 1 Institute of Medicine (US) Committee on Quality of Health Care in America. To Err is Human: building a safer health system. National Academies Press; 2000
• 2 Macias M, Bernabeu-Andreu FA, Arribas I, Navarro F, Baldominos G. Impact of a barcode medication administration system on patient safety. Oncol Nurs Forum 2018; 45 (01) E1-E13
• 3 Barakat S, Franklin BD. An evaluation of the impact of barcode patient and medication scanning on nursing workflow at a UK teaching hospital. Pharmacy (Basel) 2020; 8 (03) 148
• 4 Sittig DF, Wright A, Coiera E. et al. Current challenges in health information technology-related patient safety. Health Informatics J 2020; 26 (01) 181-189
• 5 Sittig DF, Singh H. A Sociotechnical Approach to Electronic Health Record Related Safety. Key Advances in Clinical Informatics. Academic Press; 2017: 197-216
• 6 Magrabi F, Liaw ST, Arachi D, Runciman W, Coiera E, Kidd MR. Identifying patient safety problems associated with information technology in general practice: an analysis of incident reports. BMJ Qual Saf 2016; 25 (11) 870-880
• 7 Borycki E. Trends in health information technology safety: from technology-induced errors to current approaches for ensuring technology safety. Healthc Inform Res 2013; 19 (02) 69-78
• 8 Sittig DF, Singh H. Defining health information technology-related errors: new developments since to err is human. Arch Intern Med 2011; 171 (14) 1281-1284
• 9 Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006; 13 (05) 547-556
• 10 Weiner JP, Kfuri T, Chan K, Fowles JB. “e-Iatrogenesis”: the most critical unintended consequence of CPOE and other HIT. J Am Med Inform Assoc 2007; 14 (03) 387-388 , discussion 389
• 11 Makeham M, Magrabi F, Hibbert P, Hardie R-A. Literature review and environmental scan on approaches to the review and investigation of health IT-related patient safety incidents. 2017 . Accessed December 11, 2018 at: https://www.safetyandquality.gov.au/publications-and-resources/resource-library/literature-review-and-environmental-scan-approaches-review-and-investigation-health-it-related-patient-safety-incidents
• 12 Abraham J, Kitsiou S, Meng A, Burton S, Vatani H, Kannampallil T. Effects of CPOE-based medication ordering on outcomes: an overview of systematic reviews. BMJ Qual Saf 2020; 29 (10) 1-2
• 13 Brenner SK, Kaushal R, Grinspan Z. et al. Effects of health information technology on patient outcomes: a systematic review. J Am Med Inform Assoc 2016; 23 (05) 1016-1036
• 14 Powell L, Sittig DF, Chrouser K, Singh H. Assessment of health information technology-related outpatient diagnostic delays in the US veterans affairs health care system: a qualitative study of aggregated root cause analysis data. JAMA Netw Open 2020; 3 (06) e206752
• 15 Graber ML, Siegal D, Riah H, Johnston D, Kenyon K. Electronic health record-related events in medical malpractice claims. J Patient Saf 2019; 15 (02) 77-85
• 16 Nanji KC, Rothschild JM, Salzberg C. et al. Errors associated with outpatient computerized prescribing systems. J Am Med Inform Assoc 2011; 18 (06) 767-773
• 17 Ai A, Wong A, Amato M, Wright A. Communication failure: analysis of prescribers' use of an internal free-text field on electronic prescriptions. J Am Med Inform Assoc 2018; 25 (06) 709-714
• 18 Lourenco LM, Bursua A, Groo VL. Automatic errors: a case series on the errors inherent in electronic prescribing. J Gen Intern Med 2016; 31 (07) 808-811
• 19 Slight SP, Howard R, Ghaleb M, Barber N, Franklin BD, Avery AJ. The causes of prescribing errors in English general practices: a qualitative study. Br J Gen Pract 2013; 63 (615) e713-e720
• 20 Odukoya OK, Chui MA. e-Prescribing: characterisation of patient safety hazards in community pharmacies using a sociotechnical systems approach. BMJ Qual Saf 2013; 22 (10) 816-825
• 21 Canada Health Infoway. Canadian physician survey physicians' use of digital health and information technologies in practice background and methodology. 2018 . Accessed February 24, 2019 at: https://www.infoway-inforoute.ca/en/component/edocman/3643-2018-canadian-physician-survey/view-document?Itemid=0
• 22 Warm D, Edwards P. Classifying health information technology patient safety related incidents - an approach used in Wales. Appl Clin Inform 2012; 3 (02) 248-257
• 23 Palojoki S, Mäkelä M, Lehtonen L, Saranto K. An analysis of electronic health record-related patient safety incidents. Health Informatics J 2017; 23 (02) 134-145
• 24 Magrabi F, Baker M, Sinha I. et al. Clinical safety of England's national programme for IT: a retrospective analysis of all reported safety events 2005 to 2011. Int J Med Inform 2015; 84 (03) 198-206
• 25 Fong A, Adams KT, Gaunt MJ, Howe JL, Kellogg KM, Ratwani RM. Identifying health information technology related safety event reports from patient safety event report databases. J Biomed Inform 2018; 86: 135-142
• 26 Martin G, Ghafur S, Cingolani I. et al. The effects and preventability of 2627 patient safety incidents related to health information technology failures: a retrospective analysis of 10 years of incident reporting in England and Wales. Lancet Digit Health 2019; 1 (03) e127-e135
• 27 Mitchell I, Schuster A, Smith K, Pronovost P, Wu A. Patient safety incident reporting: a qualitative study of thoughts and perceptions of experts 15 years after 'To Err is Human'. BMJ Qual Saf 2016; 25 (02) 92-99
• 28 Flemons WW, McRae G. Reporting, learning and the culture of safety. Healthc Q 2012; 15 (Spec No): 12-17
• 29 Singh H, Sittig DF. A sociotechnical framework for safety-related electronic health record research reporting: the SAFER reporting framework. Ann Intern Med 2020; 172 (11) S92-S100
• 30 Runciman WB. Lessons from the Australian Patient Safety Foundation: setting up a national patient safety surveillance system–is this the right model?. Qual Saf Health Care 2002; 11 (03) 246-251
• 31 Howell A-M, Burns EM, Hull L, Mayer E, Sevdalis N, Darzi A. International recommendations for national patient safety incident reporting systems: an expert Delphi consensus-building process. BMJ Qual Saf 2017; 26 (02) 150-163
• 32 Bellwood P, Borycki EM, Kushniruk AW. Awareness of technology-induced errors and processes for identifying and preventing such errors. Stud Health Technol Inform 2015; 208: 61-65
• 33 Digital Health Canada. e-Safety: Update 2017-2018. e-Safety Community of Action; 2018
• 34 Kaminski J. . Theory applied to informatics: socio-technical theory. Can J Nurs Inform 2022;17(3–4)
• 35 Noble DJ, Pronovost PJ. Underreporting of patient safety incidents reduces health care's ability to quantify and accurately measure harm reduction. J Patient Saf 2010; 6 (04) 247-250
• 36 Macrae C. The problem with incident reporting. BMJ Qual Saf 2016; 25 (02) 71-75
• 37 Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci 2016; 11: 38
• 38 Cochrane D, Taylor A, Miller G. et al. Establishing a provincial patient safety and learning system: pilot project results and lessons learned. Healthc Q 2009; 12 Spec No Patient: 147-153
• 39 World Health Organization. Conceptual Framework for the International Classification for Patient Safety: Version 1.1. Vol. 1. 2009
• 40 Magrabi F, Ong M-S, Runciman W, Coiera E. An analysis of computer-related patient safety incidents to inform the development of a classification. J Am Med Inform Assoc 2010; 17 (06) 663-670
• 41 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19 (Suppl 3, Suppl 3) i68-i74
• 42 Meeks DW, Takian A, Sittig DF, Singh H, Barber N. Exploring the sociotechnical intersection of patient safety and electronic health record implementation. J Am Med Inform Assoc 2014; 21 (e1): e28-e34
• 43 McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012; 22 (03) 276-282
• 44 Meeks DW, Smith MW, Taylor L, Sittig DF, Scott JM, Singh H. An analysis of electronic health record-related patient safety concerns. J Am Med Inform Assoc 2014; 21 (06) 1053-1059
• 45 Castro GM, Buczkowski L, Hafner JM. The contribution of sociotechnical factors to health information technology-related sentinel events. Jt Comm J Qual Patient Saf 2016; 42 (02) 70-76
• 46 Kang H, Wang J, Yao B, Zhou S, Gong Y. Toward safer health care: a review strategy of FDA medical device adverse event database to identify and categorize health information technology related events. JAMIA Open 2018; 2 (01) 179-186
• 47 Menon S, Singh H, Giardina TD. et al. Safety huddles to proactively identify and address electronic health record safety. J Am Med Inform Assoc 2017; 24 (02) 261-267
• 48 Laatikainen O, Miettunen J, Sneck S, Lehtiniemi H, Tenhunen O, Turpeinen M. The prevalence of medication-related adverse events in inpatients-a systematic review and meta-analysis. Eur J Clin Pharmacol 2017; 73 (12) 1539-1549
• 49 Panagioti M, Khan K, Keers RN. et al. Prevalence, severity, and nature of preventable patient harm across medical care settings: systematic review and meta-analysis. BMJ 2019; 366: l4185
• 50 de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA. The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care 2008; 17 (03) 216-223
• 51 Gens-Barberà M, Hernández-Vidal N, Vidal-Esteve E. et al. Analysis of patient safety incidents in primary care reported in an electronic registry application. Int J Environ Res Public Health 2021; 18 (17) 8941
• 52 Williams H, Edwards A, Hibbert P. et al. Harms from discharge to primary care: mixed methods analysis of incident reports. Br J Gen Pract 2015; 65 (641) e829-e837
• 53 Masotti P, McColl MA, Green M. Adverse events experienced by homecare patients: a scoping review of the literature. Int J Qual Health Care 2010; 22 (02) 115-125
• 54 Maillet É, Paré G, Currie LM. et al. Laboratory testing in primary care: a systematic review of health IT impacts. Int J Med Inform 2018; 116: 52-69
• 55 Vincent C, Carthey J, Macrae C, Amalberti R. Safety analysis over time: seven major changes to adverse event investigation. Implement Sci 2017; 12 (01) 151
• 56 Shojania KG. The frustrating case of incident-reporting systems. Qual Saf Health Care 2008; 17 (06) 400-402
• 57 Howe JL, Hettinger AZ, Ratwani RM. Using patient safety-event report data to assess health-IT safety: benefits and challenges. Lancet Digit Health 2019; 1 (03) e104-e105
• 58 Gong Y, Kang H, Wu X, Hua L. Enhancing patient safety event reporting. A systematic review of system design features. Appl Clin Inform 2017; 8 (03) 893-909
• 59 Wyatt KD, Benning TJ, Morgenthaler TI, Arteaga GM. Development of a taxonomy for medication-related patient safety events related to health information technology in pediatrics. Appl Clin Inform 2020; 11 (05) 714-724
• 60 Wang J, Liang H, Kang H, Gong Y. Understanding health information technology induced medication safety events by two conceptual frameworks. Appl Clin Inform 2019; 10 (01) 158-167
• 61 Fong A. Realizing the power of text mining and natural language processing for analyzing patient safety event narratives: the challenges and path forward. J Patient Saf 2021; 17 (08) e834-e836
• 62 Fong A, Komolafe T, Adams KT, Cohen A, Howe JL, Ratwani RM. Exploration and initial development of text classification models to identify health information technology usability-related patient safety event reports. Appl Clin Inform 2019; 10 (03) 521-527
• 63 Jacobson AF, Hamilton P, Galloway J. Obtaining and evaluating data sets for secondary analysis in nursing research. West J Nurs Res 1993; 15 (04) 483-494
• 64 Kim MO, Coiera E, Magrabi F. Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review. J Am Med Inform Assoc 2017; 24 (02) 246-250
• 65 Bonetti D, Johnston M, Pitts NB. et al. Knowledge may not be the best target for strategies to influence evidence-based practice: using psychological models to understand RCT effects. Int J Behav Med 2009; 16 (03) 287-293

Publication History

Received: 02 July 2023

Accepted: 10 October 2023

Article published online:
27 December 2023