Evaluation of Multimedia Medication Reconciliation Software: A Randomized Controlled, Single-Blind Trial to Measure Diagnostic Accuracy for Discrepancy Detection

Abstract

Background The Veterans Affairs Portland Healthcare System developed a medication history collection software that displays prescription names and medication images.

Objective This article measures the frequency of medication discrepancy reporting using the medication history collection software and compares with the frequency of reporting using a paper-based process. This article also determines the accuracy of each method by comparing both strategies to a best possible medication history.

Study Design Randomized, controlled, single-blind trial.

Setting Three community-based primary care clinics associated with the Veterans Affairs Portland Healthcare System: a 300-bed teaching facility and ambulatory care network serving Veteran soldiers in the Pacific Northwest United States.

Participants Of 212 patients with primary care appointments, 209 patients fulfilled the study requirements.

Intervention Patients randomized to a software-directed medication history or a paper-based medication history. Randomization and allocation to treatment groups were performed using a computer-based random number generator. Assignments were placed in a sealed envelope and opened after participant consent. The research coordinator did not know or have access to the treatment assignment until the time of presentation.

Main Outcome Measures The primary analysis compared the discrepancy detection rates between groups with respect to the health record and a best possible medication history.

Results Of 3,500 medications reviewed, we detected 1,435 discrepancies. Forty-six percent of those discrepancies were potentially high risk for causing an adverse drug event. There was no difference in detection rates between treatment arms. Software sensitivity was 83% and specificity was 91%; paper sensitivity was 81% and specificity was 94%. No participants were lost to follow-up.

Conclusion The medication history collection software is an efficient and scalable method for gathering a medication history and detecting high-risk discrepancies. Although it included medication images, the technology did not improve accuracy over a paper list when compared with a best possible medication history.

Trial Registration ClinicalTrials.gov Identifier: NCT02135731.

Keywords

medication reconciliation - clinical information systems - quality improvement - medication errors - medication adherence - clinical informatics - consumer health informatics - medication review

Background and Significance

Any incongruity in prescription information between two or more different information sources may result in a medication discrepancy.[1] The risk of an individual discrepancy is relatively small.[2] However, in aggregate, medication discrepancies during care transitions and clinic visits are a common cause of preventable adverse drug events (ADEs) and patient harm.[3] [4] [5] [6] [7] [8] [9] [10] [11] Unintentional discrepancies in allergy and medication information contribute to nearly half a million hospitalizations and cost the United States' (U.S.) health care system upwards of $1 billion annually.[12] [13] [14] [15]

According to the Institute of Healthcare Improvement, medication reconciliation (MR) is the “process of identifying the most accurate list of all medications a patient is taking…and using this list to provide correct medications for patients anywhere within the healthcare system.”[16] Studies show that standardized MR programs reliably identify discrepancies and reduce medical error.[5] [17] [18] [19] [20] [21] While quality advocacy organizations and regulatory agencies recommend implementing scalable systems-based solutions, health care organizations typically struggle with an array of individual- and system-based implementation barriers.[3] [16] [22] [23] [24] [25] [26] [27] [28] [29] [30]

Clinicians compiling medication histories under time pressure often adopt manual workarounds to overcome poor electronic health record (EHR) usability and lack of data interoperability.[29] [30] [31] [32] [33] They cite challenges with tight schedules, competing care priorities, and limited patient reliability.[30] This is unsurprising given that a medication history can require 20 minutes and a high-quality reconciliation can take up to 80 minutes.[33] Therefore, many clinicians dismiss MR as a set of “administrative accounting tasks” and skip reviewing medications with patients altogether.[34] By contrast, quality MR programs foster patient-centered discussion, pull data from multiple sources, and promote interdisciplinary communication.[23] [35] Clearly, there is a need for user-centered tools that assist with history collection and that improve the efficiency, consistency, and accuracy of MR.[17] [26] [27] [36] [37] [38] [39] [40] [41]

Problem Statement

The Veterans Affairs (VA) initiated an MR campaign to establish standards, promote change, and measure implementation effectiveness.[42] The campaign required clinicians to collect a medication history at every encounter and compare with facility documentation. Many clinics failed to meet these guidelines. Clinicians often found it difficult to collect or update medication information, opting instead to copy and paste existing EHR medication lists into their notes. In a previous survey of local primary care providers, respondents identified three barriers affecting MR: poor EHR design, inflexible workflows, and insufficient time.[29] We therefore developed a medication history collection software to engage patients, improve data integration, and streamline the process.[38]

We modeled our Automated Patient History Intake Device (APHID) software after interactive self-service functionality commonly featured on retail Web sites and commercial air travel check-in kiosks.[38] [43] Patients used an APHID-enabled kiosk located in the clinic waiting room to check-in for their appointments and review the names and pictures of their prescriptions. Clinicians then imported the results into the EHR for reference during the medication interview.

Although our initial piloting efforts demonstrated the viability of this approach, we had not yet measured its accuracy compared with traditional methods of history collection such as interviews and paper questionnaires.[29] [38] [43] Few studies have demonstrated the effectiveness of self-service software or multimedia tools for gathering reliable medication information.[44] [45] [46] [47] [48] [49] In small nonrandomized pilot studies, Hornick et al and Kimmel et al found that medication images improved patient recall for certain classes of medications.[50] [51] Before these strategies can be recommended, high-quality studies are needed to inform design and establish the accuracy of a self-reported medication history.

Objective

This study compared the diagnostic accuracy of the APHID collection process to a paper-based collection process. To determine which of these methods was more effective, we compared both strategies to the EHR list and a reference standard. For our reference standard, we assembled a best possible medication history (BPMH) ([Table 1]): a systematic clinician-conducted history using several information sources.[34] We hypothesized that participants using APHID would report more medication discrepancies than those using a paper list. We also hypothesized that the participant-reviewed APHID list would be more accurate than the participant-reviewed paper list when compared with the BPMH.

Table 1 Glossary of terms used in this article

Term	Definition
Adverse drug event (ADE)	Allergic reactions, adverse effects, or unintentional overdoses[6]
Automated Patient History Intake Device (APHID)	Veterans' Affairs (VA) software that supports collection and documentation of a medication history[38]
APHID list	A medication list generated by the APHID software that includes current prescriptions written at the study facility, 6 months of expired prescriptions written at the study facility, non-VA prescriptions and nonprescription medications documented at the study facility, and prescriptions written at other VA facilities[38]
Best possible medication history (BPMH)	Clinician-gathered medication history adjudicated with the participant-reviewed blinded list; reference standard[34]
Clinician-gathered medication history	A medication history collected by the clinician researcher while referencing the EHR and inspecting medications brought in by the patient
Electronic health record (EHR) list	List of prescription information in the VA electronic health record
Expired medications	Electronic prescriptions that have passed a predefined expiration date set by the EHR
Kiosk	Self-service hardware equipped with a touch-screen and installed with APHID software[38] [45]
Medication discrepancy	Any unintended incongruity between prescription lists from two sources[5]
Medication errors	Failure in prescribing or treatment process that can result in patient harm[125]
Medication history	A process of identifying the list of all medications a patient is taking by interviewing the patient/family and reviewing available documentation[9]
Medication information	Information about prescribing/administration/consumption of medications
Medication reconciliation	A process of identifying the most accurate list of all medications a patient is taking and using this list to provide correct information across the continuum of care[16]
Non-VA medications	Medications procured outside the VA by the patient and documented in the EHR by a clinician
Paper list	A medication list that includes current prescriptions written at the study facility, 6 months of expired prescriptions written at the study facility, non-VA prescriptions and nonprescription medications documented at the study facility, and prescriptions written at other VA facilities
Participant adherence	Extent to which medications are taken as directed
Participant-reviewed APHID list	APHID list of medications with participant adherence documented
Participant-reviewed blinded list	Form created by the research coordinator listing medications and participant-furnished adherence history
Participant-reviewed paper list	Paper list of medications with participant adherence documented[1]
Root cause (participant-based)	Discrepancy caused by a factor under the patient's control[1]
Root cause (system-based)	Discrepancy caused by a clinician or health system factor
VA medications (local)	Electronic prescriptions written and documented in the study facility EHR
VA medications (remote)	Electronic prescriptions written at VA facilities other than the study facility

Materials and Methods

Theory

We based our study design on two well-established theories: (1) the Systems Engineering to Improve Patient Safety (SEIPS) framework; and (2) the Pictorial Superiority Effect (PSE) ([Fig. 1]). Carayon et al's SEIPS framework argues that system constructs (i.e., people, technologies, workflow, and culture) dictate health system processes and clinical outcomes.[52] In our adaptation of the framework, MR is a cyclical macroprocess composed of linked subprocesses (e.g., history collection, data adjudication, discrepancy resolution).[53] The quality of MR can only be as effective as the initial history.[35] [53] We therefore designed our intervention to address system constructs supporting history collection.[23] [52] [53] [54]

Fig. 1 Adaptation of Carayon's systems engineering to improve patient safety (SEIPS) framework.

The PSE contends that humans encode, store, and retrieve images from memory more easily than text or auditory information.[55] [56] [57] Studies indicate patient education materials using pictures and pictograms can influence health literacy and improve comprehension.[49] [51] [58] [59] [60] [61] [62] Since it is rare for patients to bring their medications to clinic, we hypothesized that providing patients with medication images would reduce the number of errors caused by “look-alike” and “sound-alike” medications.[63] [64] [65]

Description of the Technology

Our technology consists of three main components: (1) a self-service kiosk; (2) the APHID medication history collection software; and (3) an interface to the facility EHR ([Fig. 2]).[29] [38] [43] For a complete description of the technology, please refer to our technology development and deployment manuscript.[38]

Fig. 2 Representative screenshot and output from Automated Patient History Intake Device (APHID).

The APHID software has access to all prescription information in the EHR and can use stored metadata to match each medication with a digital image. Our organization manages the medication supply chain for most prescriptions; medications written by VA prescribers are typically dispensed from VA facilities or regional mail-out distribution centers. The EHR stores medication dispense dates, prescription refill histories, medication inventory numbers, and U.S. National Drug Code (NDC) numbers. The NDC numbers are unique 10-digit, 3-segment numbers assigned by the U.S. Food and Drug Administration to all drugs distributed in the United States. APHID uses a combination of NDC numbers and dispensing data to match an image with each prescription.[66] To assemble a medication list for patient review, APHID retrieves prescription data from all VA facilities and pairs each medication name and instructions for use with a single digital photograph of the prescription. If a patient has received several prescriptions for the same drug, APHID uses the image associated with the last dispense date.

It is customary for VA clinicians to document prescriptions from non-VA practitioners and nonprescription medications reported by the patient. Approximately 10 to 30% of prescriptions are procured outside the VA. APHID will display this information when available in the EHR. However, the VA does not exchange health data with community drug dispensaries. Thus, APHID cannot match images to prescriptions procured outside the VA.

Subjects and Settings

We recruited U.S. Veteran patients from three primary care clinics associated with the VA Portland Healthcare System (VAPORHCS). VHAPORHCS is a 300-bed tertiary care hospital with eight associated ambulatory care centers located in northern Oregon and southern Washington states. It is part of the Veterans Health Affairs, a nationwide care network of over 150 hospitals.[67] [68] Most Veterans are male (94%), older than the average civilian patient, have a greater number of medical comorbidities, and use more medications.

Study Design

We conducted a prospective parallel-randomized, controlled, single-blind study of the APHID software. Using the BPMH as a reference standard, we compared the discrepancy counts reported using a paper list with those reported by APHID ([Fig. 3]). We designed our study as per the CONSORT guidelines for reporting clinical trials and STARD guidelines for reporting studies of diagnostic tests.[69] [70] [71] For a detailed description of the methods and instruments, please refer to our previously published protocol.[53]

Fig. 3 Protocol for Trial.

From June 2009 through December 2011, a research coordinator contacted all patients scheduled in participating clinics and screened them for inclusion in the study. Patients were eligible for inclusion if they were over the age of 18 years, taking three or more medications, and had completed at least one appointment in the past. Exclusion criteria included: the inability to read or speak English; the presence of cognitive impairment or mental illness; visual impairment; and physical impairment that might preclude use of a mouse or keyboard. The research coordinator asked participating patients to bring in all their medications to the appointment.

On the day of the study appointment, the research coordinator confirmed the study participant's eligibility and then consented, randomized, and assigned the participant to one of the two treatments ([Fig. 3], step 2). A member of the research team determined the treatment assignment using a computer-based random number generator and placed the output in a sealed envelope. The research coordinator did not know or have access to the treatment assignment until the time of presentation when the envelope was opened.

The research coordinator asked participants to review, correct, and amend their medication lists ([Fig. 3], step 3). Participants in the intervention arm used APHID at a standard computer workstation, whereas participants in the control arm used paper on a clipboard. Participants using APHID completed an onscreen questionnaire asking them about (1) their prescriptions dispensed by the study facility; (2) other VA prescriptions; (3) nonprescription medications; and (4) 6 months of expired prescriptions. Each prescription was displayed, one at a time, with an image where available. Participants were asked to indicate adherence using one of four structured response buttons (“Yes, taking as written above; No, taking differently; No, NOT taking; Unsure”). Participants then added any new or missing medications.

Participants in the control arm reviewed a paper list that included (1) prescriptions dispensed by the study facility; (2) other VA prescriptions; (3) nonprescription medications; and (4) 6 months of expired prescriptions. This control included more information than what was available in the facility EHR and was more stringent than the “usual care” practice of importing an automated list into clinic notes. We selected this control to isolate the effects of medication pictures and self-service software.[17] [23] [24] [33] [37] [67] [72] [73] [74] It was crucial that the paper and APHID lists contained the same medications for review and the same number of opportunities to report a discrepancy.

The research coordinator asked the control group participants to place a “yes” on the sheet next to medications they were taking and “no,” “differently,” or “unsure” next to medications that they were not taking, taking differently, or had a question about. We coded any response other than “yes” as a discrepancy for active medications. We coded any response other than “no” as a discrepancy for expired medications. Participants listed additional medications at the bottom of the form. We could not blind participants to treatment status. The research coordinator then transcribed participant responses onto a paper form (i.e., participant-reviewed blinded list) to mask treatment assignment from the researcher completing the interview ([Fig. 3]; step 4).

Clinician researchers (an internal medicine physician, a clinical pharmacist, and an advanced practice nurse) were trained to collect a medication history using an interview script. The researchers then independently interviewed standardized patients to eliminate variation in interview technique. The performance characteristics were identical between researchers after two practice cycles.

In the study, a blinded clinician researcher met with each participant and completed a medication history using the interview script ([Fig. 3]; step 5). This is the most common method used in MR studies to establish a reference standard.[13] [17] [18] [67] [75] [76] [77] [78] [79] The researchers were also instructed to review the EHR and inspect the medication containers. Then the researcher recorded a discrepancy on a spreadsheet if the participant was not taking a medication associated with a current prescription, taking a medication associated with an expired prescription, taking a medication differently than instructed, or taking a new medication.

Our reference standard (i.e., the BPMH) included an additional step and an additional information source ([Fig. 3], step 6). The BPMH included the clinician-gathered medication history, a review of EHR pharmacy records, a prescription vial or direct pill inspection, and a double-check using the participant-reviewed blinded list. The research coordinator furnished the clinician researcher with the blinded list. The clinician then completed the “double-check” with the patient, adjudicating mismatches identified between the clinician history and the blinded list. The researcher then furnished the BPMH to the primary care provider.

Our institutional review board (IRB) required the adjudication step ([Fig. 3], step 6) for safety purposes; it was crucial to disclose all information sources to the primary care team. Furthermore, we used both histories—the clinician-gathered medication history and the BPMH—during statistical analysis to screen for differences or trends between the first and second clinician review.

Researchers classified all discrepancies by descriptive type using a typology adapted from Pippins et al.[80] Researchers also classified discrepancies by root cause using an instrument adapted from Orrico and Smith et al.[1] [81] A blinded clinician panel then assigned risk scores using a classification scheme adapted from Pippins et al and Wong et al (see [Appendix Fig. A1] for risk assessment protocol)[80] [82] ([Fig. 3], step 7). We used sample sets of discrepancies collected during piloting to train clinician raters on the instrument and calibrate responses.

Analysis

To determine discrepancy rates, we compared the participant-reviewed lists against the EHR list and then calculated the proportion of medications with a discrepancy ([Fig. 3], step 1–step 2). Similarly, we compared the participant-reviewed lists against the BPMH and calculated the proportion of medications with a discrepancy ([Fig. 3], step 2–step 6). The primary outcome measure was the difference in discrepancy rates between each arm with respect to the EHR list. Our secondary outcome measures were the difference in discrepancy rates and high-risk discrepancy rates between each arm compared with the BPMH. Assuming a discrepancy base rate of 25% per participant list, our power calculations indicated that we needed a sample size of 210 participants to detect a difference of 15% in detection rates between treatment arms.[43] [53] We used a crossed random-effects model (three phases of evaluation crossed with participant lists) to account for discrepancy-clustering effects at the participant level. Discrepancy rates were calculated for two sets of comparisons: (1) treatment arm versus EHR; and (2) treatment arm versus BPMH. We completed a poststratified analysis to compare differences in discrepancy counts between treatment arms correcting for the number of medications between treatment arms (i.e., opportunities to detect an error). We also computed intercorrelation coefficients (ICCs) of absolute agreement by treatment arm for individual discrepancies at the level of the medication list item across evaluation phases (clinician-gathered medication history and BPMH) to assess whether one of the treatment arms showed significantly more agreement than the other with respect to a given medication list item. No gross difference in agreement was seen between the treatment arms.

We assessed the accuracy of treatment methods relative to the BPMH by comparing the discrepancy status (either “yes” or “no” according to the BPMH within each treatment arm) of each medication list item that was identified by both the BPMH and the treatment. We then tallied the counts of (“yes,” “yes”), (“yes,” “no”), (“no,” “yes”), and (“no,” “no”), where the first response in each pair was from the BPMH and the second response was from a treatment method. This yielded two sets of four counts each; arranged in two 2 × 2 cross-tabulation tables and individually analyzed on standard diagnostic agreement metrics. We assumed that the BPMH represented the true status and the treatment method represented the test status. The metrics we report include sensitivity (fraction of test positives among true positives), specificity (fraction of test negatives among true negatives), positive predictive value (fraction of true positives among test positives), negative predictive value (fraction of true negatives among test negatives), positive likelihood ratio (sensitivity divided by false positive fraction among true negatives), and negative likelihood ratio (false negative fraction among true positives, divided by specificity), where the likelihood ratios are weighted by the prevalence odds. Confidence intervals were calculated for each metric on each cross-tabulation using exact binomial distributions.

Results

We assessed 614 patients for eligibility; 220 participants were enrolled and randomized for the study ([Fig. 4]). The study pilot involved eight participants and three participants were withdrawn, leaving 209 participants included in the final analysis. No participants were lost to follow-up. There were no incidents of accidental unblinding treatment status for the clinician interviewers.

Fig. 4 Flowchart for patient enrollment, randomization, and analysis.

Participant sample characteristics and descriptive statistics are reported in [Table 2]. Most enrolled participants were male, with a mean age of 66.5 years, and an estimated 5.6 chronic medical illnesses. Each participant had an average of 11.5 active prescriptions. When accounting for recently expired and nonprescription medications, we reviewed an average of 16.7 medications per participant.

Table 2 Descriptive statistics of participant sample

Characteristic	Control		Intervention		Total
Participants, count (%)	102	(48.8)	107	(51.2)	209	(100.0)
Male gender, count (%)	99	(97.1)	101	(94.4)	200	(95.7)
Age (y), mean (SD)	67.6	(12.3)	65.5	(12.2)	66.5	(12.3)
Chronic conditions, mean (SD)	5.5	(2.0)	5.7	(2.1)	5.6	(2.1)
Medications, mean (SD)	16.8	(7.1)	16.7	(8.5)	16.7	(7.8)
Current prescriptions in EHR	11.5	(5.7)	11.5	(6.0)	11.5	(5.9)
Expired prescriptions in EHR	2.5	(2.4)	2.9	(3.1)	2.7	(2.8)
Newly reported medications not in EHR	2.9	(2.6)	2.2	(2.7)	2.5	(2.7)
Education, count (%)
Less than 12th grade	6	(5.9)	10	(9.3)	16	(7.7)
High school graduate	26	(25.5)	22	(20.6)	48	(23.0)
Some college (no degree)	26	(25.5)	23	(21.5)	49	(23.4)
College degree	44	(43.1)	52	(48.6)	96	(45.9)

Abbreviations: EHR, electronic health record; SD, standard deviation.

Descriptive statistics for all discrepancies detected are outlined in [Table 3]. Using all information sources, our team identified 3,500 medications and 1,435 discrepancies. We detected 530 discrepancies using the paper list and 594 discrepancies using the APHID list. Of the 1,435 discrepancies identified by the BPMH, 657 (46%) were high or very high risk. We traced 47% of the discrepancies to a system-based root cause (i.e., clinical documentation errors) and the remainder to a participant-based root cause (e.g., nonadherence). VA medications accounted for 49% of the discrepancies by prescription status, expired VA medications accounted for 15%, and medications not in the EHR accounted for 37%. We did not identify any differences between treatment arms.

Table 3 Descriptive statistics for all discrepancies identified using any method (N = 3,500)

Classification	Paper count (%)		APHID count (%)		BPMH count (%)
Total medication list items	1,717		1,783		3,500
Total discrepancies detected	530		594		1,435
Discrepancies sorted by potential ADE risk[a]
High or very high risk	244	(46.0)	298	(50.2)	657	(45.8)
Low or medium risk	284	(53.6)	296	(49.8)	775	(54.0)
Missing risk evaluation	2	(0.4)	0	(0.0)	3	(0.2)
Discrepancies sorted by root cause
System-based root cause	224	(42.3)	259	(43.6)	674	(47.0)
Participant-based root cause	306	(57.7)	335	(56.4)	761	(53.0)
Discrepancies sorted by prescription status
VA medications	271	(51.1)	326	(54.9)	696	(48.5)
Expired VA medications	97	(18.3)	136	(22.9)	211	(14.7)
Medications not in EHR	162	(30.6)	132	(22.2)	528	(36.8)
Discrepancies sorted by prescription source
Local VA facility	294	(55.5)	359	(60.4)	764	(53.2)
Remote VA facility	2	(0.4)	30	(5.1)	32	(2.2)
Non-VA	234	(44.2)	205	(34.5)	639	(44.5)
Discrepancies sorted by classification
Omission	256	(48.3)	253	(42.6)	705	(49.1)
Commission	188	(35.5)	219	(36.9)	415	(28.9)
Dose	36	(6.8)	45	(7.6)	128	(8.9)
Frequency	46	(8.7)	75	(12.6)	173	(12.1)
Substitution	4	(0.8)	2	(0.3)	14	(1.0)

Abbreviations: ADE, adverse drug event; APHID, Automated Patient History Intake Device; BPMH, best possible medication history; EHR, electronic health record; VA, Veterans Affairs.

Note: A total of 253 medications were only identified by the BPMH and are reflected in the total count. All percentages represent percent of discrepancies within the treatment group.

^a Clinicians did not have enough contextual information to confidently assign a risk category.

There were no statistically significant differences in the rate of discrepancies reported (i.e., primary outcome) for each treatment arm when compared with the study facility EHR ([Table 4]). An average of 35% (0.35 ± 0.20) of medications on each list included a discrepancy (p = 0.89); 15% of all medications in the control arm and 17% of all medications in the intervention arm included a high-risk discrepancy (this corresponds to 43% of all discrepancies in the control group and 49% of all discrepancies in the intervention group).

Table 4 Disagreement between the EHR list, treatment lists, and the reference standard

Comparison	Paper (102 participants)		APHID (107 participants)		Raw difference	p-Value
Treatment versus EHR # medications	1,574		1,673		99
Proportion of list with discrepancies
Total discrepancies, mean (SD)	0.35	(0.20)	0.35	(0.19)	0.00	0.89
High and very high risk, mean (SD)	0.15	(0.13)	0.17	(0.15)	0.02	0.34
System-based, mean (SD)	0.14	(0.14)	0.15	(0.13)	0.01	0.64
Participant-based, mean (SD)	0.21	(0.21)	0.20	(0.16)	0.01	0.64
Number of discordant lists – count (%)	99	(97)	104	(97)	5	1.00
Treatment versus BPMH # medications	1,574		1,673		99
Proportion of list with discrepancies
Total discrepancies, mean (SD)	0.13	(0.12)	0.13	(0.13)	0.00	0.90
High and very high risk, mean (SD)	0.04	(0.07)	0.05	(0.07)	0.01	0.67
System-based discrepancies, mean (SD)	0.04	(0.06)	0.04	(0.06)	0.00	0.97
Participant-based discrepancies, mean (SD)	0.06	(0.09)	0.05	(0.07)	0.01	0.31
Number of discordant lists, count (%)	78	(76)	76	(71)	2	0.43
EHR versus BPMH # medications[a]	1,717		1,783		66
Proportion of list with discrepancies
Total discrepancies, mean (SD)	0.43	(0.20)	0.39	(0.17)	0.04	0.12
High and very high risk	0.19	(0.14)	0.18	(0.14)	0.00	0.67
System-based	0.20	(0.17)	0.17	(0.14)	0.03	0.24
Participant-based	0.23	(0.20)	0.22	(0.17)	0.01	0.55
Discordant lists, count (%)	101	(99)	106	(99)	5	1.00

Abbreviations: APHID, Automated Patient History Intake Device; BPMH, best possible medication history; EHR, electronic health record; SD, standard deviation.

Note: Discrepancies are reported as the proportion of medications per list with an error. Statistics calculated using the participant as the unit of analysis. N = 209 participants; 3,500 medications.

^a The BPMH identified 253 additional medications not on the EHR or in either treatment.

When comparing treatment arms to the BPMH (i.e., secondary outcome), we did not identify any differences in discrepancy rates ([Table 4]). The BPMH identified an additional 253 medications not recorded in the EHR or by either treatment arm. The BPMH included ∼13% (0.13 ± 0.13) more discrepancies than either treatment; 31 to 38% of those discrepancies (4–5% of all medications reviewed) were high risk. As a test for bias (i.e., an interaction between test and reference standard), we compared detection rates for the clinician-gathered medication history with the BPMH. We calculated a concordance rate of 98.6% (kappa = 0.97), arguing against any interaction between the test and the reference standard. When comparing the EHR lists to the BPMH, 207 of the 209 charts (99%) included one or more discrepancies.

[Table 5] lists the cross-tabulations for each treatment arm compared with the BPMH. Cross-tabulations only included medications present in the treatment list and the BPMH. New medications only identified on the BPMH were undefined for the treatment arms and could not be counted as true positives or false positives. The paper-based process had a sensitivity of 81% and APHID had a sensitivity of 83%. The paper-based process had a specificity of 94% and APHID had a specificity of 91%. Overall, we did not detect a statistically significant difference in the accuracy of either method for detecting discrepancies or high-risk discrepancies when compared with the BPMH.

Table 5 Discrepancy reporting accuracy for each treatment

		Control (paper)		Intervention (APHID)
		Discrepancy (+)	No discrepancy (–)	Discrepancy (+)	No discrepancy (–)
BPMH	Discrepancy	474	108	500	100
	No discrepancy	56	936	94	979
		530		594
		Total = 1,574 medications		Total = 1,673 medications
Validity measure		Ratio (95% confidence interval)
Sensitivity		81 (78, 85)		83 (80, 86)
Specificity		94 (93, 96)		91 (89, 93)
Positive predictive value		89 (87, 92)		84 (81, 87)
Negative predictive value		90 (88, 91)		91 (89, 92)
Positive likelihood ratio[a]		8.46 (6.60, 10.86)		5.32 (4.40, 6.42)
Negative likelihood ratio[a]		0.12 (0.10, 0.14)		0.10 (0.08, 0.12)

Abbreviations: APHID, Automated Patient History Intake Device; BPMH, best possible medication history.

Note: An additional 253 medications were identified by the BPMH that were not identified in either treatment arm. Item-wise comparison cannot be calculated for medications not defined in both methods; they are not represented in the accuracy calculation. Percents rounded to nearest percentile. N = 3,247 medications.

^a Likelihood ratios weighted by prevalence.

Discussion

Principal Findings

Discrepancy detection rates were similar between the paper-based and APHID processes for all dimensions of analysis. The addition of medication images did not affect history accuracy. Discrepancy rates, sensitivity statistics, and negative predictive values were virtually identical when compared with the EHR, even after adjusting for participant characteristics, discrepancy risk category, or root cause.

APHID offers an efficient and patient-centered method for collecting a medication history and documenting discrepancies. It compiled a list of medications from across the VA enterprise and helped identify nearly 90% of all discrepancies in our sample. APHID's detection rates compared favorably with best-practice interviews (i.e., ≥ 1 discrepancy in 99% of all EHR lists). In all, over a third of the prescriptions included a discrepancy, nearly half of which were high-risk. Approximately 15% of prescriptions with discrepancies were expired, indicating that asking about expired prescriptions identified errors of omission that might otherwise have gone undetected. This illustrates the value of using a patient-centered and standardized process to collect a medication history.[50] [67] [68] [73] [83]

Although the self-service history was more complete that the EHR list, the BPMH identified an additional high-risk discrepancy in nearly 5% of medications reviewed. This demonstrates the importance of clinician engagement. Self-service history collection techniques work in conjunction with—not in lieu of—a clinician-mediated medication history.[41] [46] [68] [73] [84]

Several possibilities may explain why our study detected more discrepancies than most other MR studies published in the last 5 years (34–88%).[25] [27] [37] [46] [47] [72] [85] [86] First, Veterans tend to be older, more medically complex, and take more medications than the general population. Each of these factors has been shown to correlate with the incidence of medication errors.[74] [76] [87] [88] [89] [90] Second, sequential history collection steps may improve patient recall. Studies suggest single histories may be less reliable, prone to drift, and overestimate compliance by up to 20%.[76] [77] [87] [91] [92] [93] [94] Third, studies show that data gathered from self-administered questionnaires are less affected by social desirability bias than interviews;[95] patients may be more likely to report nonadherence when independently correcting a medication list. Compared with pill counts, interviews have a reported sensitivity of 55 to 80% and a specificity of 70 to 87%.[77] [93] Therefore, our self-reporting techniques may have been more effective than a typical interview. Fourth, our BPMH used a combination of supply chain metadata, local prescription lists, and patient furnished data.[25] [27] [96] Combining data sources increases the ability to document “ground truth”—an important consideration for reconciliation systems.[34] [74] [76] [83] [90]

This study underscores the importance of sociotechnical fit when implementing a reconciliation program. Clinician engagement, standardized processes, and patient-centered strategies may have greater influence upon the accuracy, quality, and overall success of MR than any specific technology.[85] Most MR methods—including paper—may be equifinal if informaticians optimize workflow and implementation climate (i.e., culture, leadership, and education). Nonetheless, facilities contending with staff shortages, high patient volumes, or large geographic areas may consider using patient-centered technologies such as kiosks, secure messaging, or online portals to improve efficiency and scalability without sacrificing effectiveness or accuracy.[27] [29] [97]

Study Strengths and Limitations

To the best of our knowledge, there are very few published studies comparing MR strategies and only four other randomized controlled trials (RCTs) conducted in the ambulatory setting, none of which adhere to the CONSORT/STARD guidelines.[19] [21] [47] [74] [85] [86] [96] [98] [99] [100] Systematic reviews of ambulatory MR programs suggest more research is needed to prove the clinical impact of MR.[21] [27] [74] [96] [98] [101] Our study makes an important contribution to the literature by providing accuracy statistics for multiple collection strategies.

This study has several limitations that may affect the validity of our findings. First, the high number of reported medications, older age, and unbalanced gender distribution of our sample likely affected the results. Our findings may not be generalizable to other settings and the prevalence of medication discrepancies may be lower in other populations.[43] [102] Second, limitations in health information exchange and EHR interoperability prevented us from matching images to nonprescription and non-VA medications (∼27% of medications). This may have caused a type II error. Finally, to provide equal opportunity for discrepancy detection in each treatment arm, we compiled medication information from other VA facilities and generated longer medication lists than those accessible in our local EHR. This may have diluted our ability to identify a treatment effect.

Implications for Future Research and Policy Development

Several hypotheses may account for the apparent absence of a PSE (i.e., the phenomenon where pictures are more likely to be remembered than words). First, we could not match images to non-VA and nonprescription medications. Second, the prescriptions and images were shown on screen, one medication at a time. The similarity in physical appearance of drugs manufactured and distributed in the United States may hamper correct identification. Showing the complete list on one screen might have helped participants disambiguate similar appearing drugs. Third, there is mixed evidence to suggest that the strength of the PSE decays with age.[103] Finally, the use of computer technology may have caused an interference effect. It is critical that future consumer informatics research using pictures and visual displays isolate the design attributes that reliably promote attention, comprehension, and recall.

MR is a complex adaptive system that demands equally adaptive technologies to support practitioner workflow.[22] [34] [35] [104] [105] [106] Commensurate effort should be applied to interface usability, data interoperability, and clinician decision support.[26] [27] [46] [84] [107] [108] [109] Federated health systems and managed care enterprises can promote these efforts by spearheading use of semantically rich medication terminologies such as Rx Norm. They can also help by enforcing EHR interoperability standards and funding regional health organizations.[35] [110]

We believe our findings also emphasize the need to invest further in consumer informatics tools that engage patients and address U.S. EHR “meaningful use” standards.[111] [112] Stage 2 of the Health Information Technology for Economic and Clinical Health (HITECH) Act includes MR as one of the 25 criteria for functionality and Stage 3 expands scope to incorporate patient-entered data. These expectations, while laudable, may further stress systems, inadvertently incentivizing organizations to implement less effective solutions such as interruptive reminders or boilerplate templates. Personal health records, mobile devices, secure messaging, and other consumer-driven technologies may provide time-sensitive alternatives to engage patients and collect information that might otherwise be skipped during a busy clinic visit. The effectiveness of these hi-tech solutions can be augmented by modest interventions focusing upon purposeful interface design, patient education best-practices, and provider interviewing strategies.[11] [15] [25] [26] [44] [45] [46] [47] [60] [87] [94] [113] [114] [115] [116] [117] [118] [119] [120] Finally, patient-centered assistive technologies like smart pillboxes, wearable devices, and Internet-enhanced living environments can improve the fidelity of our data streams.[121] [122]

Conclusion

Our study suggests that gathering patient-generated data using EHR-based technologies or pen-and-paper processes can be equally effective in supporting MR. We believe the technology and workflow described herein offer a practical, safe, and scalable method to foster collaboration between patients and care teams. We have offered a strategy that combines EHR technology, business process reengineering, and patient-generated data to augment traditional history collection and improve patient engagement.[45] [47] [87] [91] [94] [123] Future research should study how to promote MR technology adoption, improve patient self-reporting, and optimize use in specialty care settings.[124]

Clinical Relevance Statement

Consumer informatics technology, such as self-service kiosks, offers a workflow-compatible solution to collect an accurate medication history and satisfy the Stage 2 Meaningful Use criteria. This randomized controlled trial shows that the patient-facing medication reconciliation software, when thoughtfully implemented using a systems engineering approach, can incorporate patient-furnished data and substantially improve discrepancy detection as compared with usual care. A variety of data collection strategies may be equifinal and further usability research is needed to understand how to effectively use medication images in consumer-facing interfaces.

Multiple Choice Questions

1. What do results of this randomized controlled trial indicate about self-service medication history collection software?

• The inclusion of medication images significantly improves patient accuracy when compared with a reference standard

• Kiosk technology does not perform as well as more traditional data collection methods such as distribution of paper-based questionnaires

• Patient-facing self-service technologies tend to be less accurate than usual care clinician-conducted interviews

• The accuracy of standardized methods for medication history collection including self-service kiosks and paper questionnaires are comparable when compared with a reference standard

  Correct Answer: The correct answer is option d. This study did not demonstrate a statistically significant difference in the performance characteristics of the software as compared with a paper-based control that did not include images. Rather, discrepancy rates, sensitivity statistics, and negative predictive values were virtually identical when compared with the reference standard. It would not be completely accurate to say the technology was less effective than a clinician-mediated history since the software's diagnostic performance was at least comparable—if not superior to—that reported for clinician-conducted histories.

2. Which statement best describes the distribution of medication history errors reported in this study?

• System-based sources of medication discrepancies are more common than patient nonadherence or recall errors

• An estimated 35 to 40% of collected medication histories include one or more discrepancies

• Approximately 15% of medication discrepancies are associated with expired prescriptions that the patient is still taking

• Between 22 and 25% of all medication discrepancies were classified as high-risk by a panel of blinded clinician raters

  Correct Answer: The correct answer is option c. Nearly 49% of discrepancies were associated with current prescriptions; 15% were associated with expired medications, and 37% were newly identified over-the-counter medications. Slightly more than half of the errors (an estimated 53%) had a patient-based root cause such as nonadherence. Note that 41% of all medications reviewed were associated with a discrepancy and 99% of medication histories included one or more discrepancies when compared with the reference standard. Over 45% of the discrepancies detected were rated as high or very high risk.

3. When implementing medication history collection software as part of an organizational medication reconciliation strategy, what factors are likely to improve performance?

• The software is primarily effective in health systems where regional health information exchange is available

• The software should include patient-facing affordances that improve patient reliability including medication images, plain language, and simple response controls

• The implementation team should pay attention to the sociotechnical fit of the product including workflow compatibility and implementation climate

• The software should be used primarily in clinical settings where resource constraints preclude collection of a more time-intensive clinician-mediated history

  Correct Answer: The correct answer is option c. This study seems to suggest that a variety of data collection methods and technologies can produce a high-quality history. Clinician engagement, standardized process, and patient-centered models may have a greater influence upon the accuracy and success of MR than any specific approach. While the availability of regional data exchange could certainly improve the quality of the history, this study shows that even without a data exchange network, the information gathered may considerably improve upon usual care. Unfortunately, the study did not show that the patient-facing interface or the inclusion of images conferred any additional benefit beyond a simple medication list. The accuracy of each treatment arm was at least comparable to the published statistics associated with a clinician-mediated history. Since results of this study suggest that combining several data sources substantially improves accuracy, a standardized patient-driven history collection method should be combined with a clinician-mediated history.

Appendix Fig. A1 Medication discrepancy risk scoring tool.

Conflict of Interest

The authors report no conflicts of interest in the research; there are no plans to commercialize the software.

Acknowledgments

The authors wish to acknowledge the generous support from the VA Portland Healthcare System and the VA Chief Business Office. The authors also thank Richard Pham, PharmD, for assistance with the development and piloting our discrepancy classification instrument.

Authors' Contributions

All the authors contributed equally to the study.

Protection of Human and Animal Subjects

This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. Our local Institutional Review Board (IRB) approved this study. All participants provided written consent. We registered with ClinicalTrials.Gov (Identifier: NCT02135731) prior to data analysis. This study was funded in part by the VA National Center for Patient Safety.

• References

• 1 Orrico KB. Sources and types of discrepancies between electronic medical records and actual outpatient medication use. J Manag Care Pharm 2008; 14 (07) 626-631
• 2 Boockvar KS, Livote EE, Goldstein N, Nebeker JR, Siu A, Fried T. Electronic health records and adverse drug events after patient transfer. Qual Saf Health Care 2010; 19 (05) e16
• 3 Institute of Medicine (IOM). Preventing Medication Errors. In: Aspden P, Wolcott J, Bootman JL, Cronenwett LR. , eds. Washington, DC: The National Academies Press; 2006
• 4 Institute for Healthcare Improvement, ed. How-to-Guide: Prevent Adverse Drug Events with Medication Reconciliation. Cambridge: Institute for Healthcare Improvement; 2011
• 5 Boockvar KS, Blum S, Kugler A. , et al. Effect of admission medication reconciliation on adverse drug events from admission medication changes. Arch Intern Med 2011; 171 (09) 860-861
• 6 Budnitz DS, Lovegrove MC, Shehab N, Richards CL. Emergency hospitalizations for adverse drug events in older Americans. N Engl J Med 2011; 365 (21) 2002-2012
• 7 Gandhi TK, Weingart SN, Borus J. , et al. Adverse drug events in ambulatory care. N Engl J Med 2003; 348 (16) 1556-1564
• 8 Varkey P, Cunningham J, O'Meara J, Bonacci R, Desai N, Sheeler R. Multidisciplinary approach to inpatient medication reconciliation in an academic setting. Am J Health Syst Pharm 2007; 64 (08) 850-854
• 9 Tam VC, Knowles SR, Cornish PL, Fine N, Marchesano R, Etchells EE. Frequency, type and clinical importance of medication history errors at admission to hospital: a systematic review. CMAJ 2005; 173 (05) 510-515
• 10 Bates DW, Cullen DJ, Laird N. , et al; ADE Prevention Study Group. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. JAMA 1995; 274 (01) 29-34
• 11 Wittich CM, Burkle CM, Lanier WL. Medication errors: an overview for clinicians. Mayo Clin Proc 2014; 89 (08) 1116-1125
• 12 Aspden P, Wolcott J, Bootman JL, Cronenwett LR. , eds. Preventing Medication Errors: Quality Chasm Series. Washington, DC: National Academies Press; 2006
• 13 Gleason KM, McDaniel MR, Feinglass J. , et al. Results of the Medications at Transitions and Clinical Handoffs (MATCH) study: an analysis of medication reconciliation errors and risk factors at hospital admission. J Gen Intern Med 2010; 25 (05) 441-447
• 14 Hasan S, Duncan GT, Neill DB, Padman R. Automatic detection of omissions in medication lists. J Am Med Inform Assoc 2011; 18 (04) 449-458
• 15 Porcelli PJ, Waitman LR, Brown SH. A review of medication reconciliation issues and experiences with clinical staff and information systems. Appl Clin Inform 2010; 1 (04) 442-461
• 16 Institute for Healthcare Improvement. Medication reconciliation to prevent adverse drug events: Institute for Healthcare Improvement; 2014 [updated 2014]. Available at: http://www.ihi.org/explore/ADEsMedicationReconciliation/Pages/default.aspx . Accessed March 8, 2014
• 17 Mueller SK, Sponsler KC, Kripalani S, Schnipper JL. Hospital-based medication reconciliation practices: a systematic review. Arch Intern Med 2012; 172 (14) 1057-1069
• 18 Schnipper JL, Hamann C, Ndumele CD. , et al. Effect of an electronic medication reconciliation application and process redesign on potential adverse drug events: a cluster-randomized trial. Arch Intern Med 2009; 169 (08) 771-780
• 19 Nassaralla CL, Naessens JM, Chaudhry R, Hansen MA, Scheitel SM. Implementation of a medication reconciliation process in an ambulatory internal medicine clinic. Qual Saf Health Care 2007; 16 (02) 90-94
• 20 Pronovost P, Weast B, Schwarz M. , et al. Medication reconciliation: a practical tool to reduce the risk of medication errors. J Crit Care 2003; 18 (04) 201-205
• 21 Krska J, Cromarty JA, Arris F. , et al. Pharmacist-led medication review in patients over 65: a randomized, controlled trial in primary care. Age Ageing 2001; 30 (03) 205-211
• 22 Kaboli PJ, Fernandes O. Medication reconciliation: moving forward. Arch Intern Med 2012; 172 (14) 1069-1070
• 23 Greenwald JL, Halasyamani L, Greene J. , et al. Making inpatient medication reconciliation patient centered, clinically relevant and implementable: a consensus statement on key principles and necessary first steps. J Hosp Med 2010; 5 (08) 477-485
• 24 Developed through the ASHP Council on Pharmacy Practice and approved by the ASHP Board of Directors on April 13, 2012, and by the ASHP House of Delegates on June 10, 2012. ASHP statement on the pharmacist's role in medication reconciliation. Am J Health Syst Pharm 2013; 70 (05) 453-456
• 25 Dullabh PM, Sondheimer NK, Katsh E, Evans MA. How patients can improve the accuracy of their medical records. eGEMs (Generating Evidence & Methods to improve patient outcomes); 2014 ; 2(3):19. Available at: https://egems.academyhealth.org/articles/abstract/10.13063/2327-9214.1080/
• 26 Bassi J, Lau F, Bardal S. Use of information technology in medication reconciliation: a scoping review. Ann Pharmacother 2010; 44 (05) 885-897
• 27 Bayoumi I, Howard M, Holbrook AM, Schabort I. Interventions to improve medication reconciliation in primary care. Ann Pharmacother 2009; 43 (10) 1667-1675
• 28 Clay BJ, Halasyamani L, Stucky ER, Greenwald JL, Williams MV. Results of a medication reconciliation survey from the 2006 Society of Hospital Medicine national meeting. J Hosp Med 2008; 3 (06) 465-472
• 29 Lesselroth BJ, Holahan PJ, Adams K. , et al. Primary care provider perceptions and use of a novel medication reconciliation technology. Inform Prim Care 2011; 19 (02) 105-118
• 30 Boockvar KS, Santos SL, Kushniruk A, Johnson C, Nebeker JR. Medication reconciliation: barriers and facilitators from the perspectives of resident physicians and pharmacists. J Hosp Med 2011; 6 (06) 329-337
• 31 Grossman JM, Gourevitch R, Cross DA. Hospital Experiences Using Electronic Health Records to Support Medication Reconciliation. Research Brief. Washington, DC: National Institute for Health Care Reform; 2014
• 32 Poon EG, Blumenfeld B, Hamann C. , et al. Design and implementation of an application and associated services to support interdisciplinary medication reconciliation efforts at an integrated healthcare delivery network. J Am Med Inform Assoc 2006; 13 (06) 581-592
• 33 Bruce Bayley K, Savitz LA, Maddalone T, Stoner SE, Hunt JS, Wells R. Evaluation of patient care interventions and recommendations by a transitional care pharmacist. Ther Clin Risk Manag 2007; 3 (04) 695-703
• 34 Fernandes O, Shojania K. Medication reconciliation in the hospital: what, why, where, when, who and how?. Healthc Q 2012; 15 (Spec No): 42-49
• 35 Rose AJ, Fischer SH, Paasche-Orlow MK. Beyond medication reconciliation: the correct medication list. JAMA 2017; 317 (20) 2057-2058
• 36 Samarth A, Grant E. Using Health Information Technology to Perform Medication Reconciliation: Findings from the AHRQ Health IT Portfolio. In: Quality AfHRa, ed. Rockville, MD: AHRQ National Resource Center for Health Information Technology; 2010: 5
• 37 Kwan JL, Lo L, Sampson M, Shojania KG. Medication reconciliation during transitions of care as a patient safety strategy: a systematic review. Ann Intern Med 2013; 158 (5 Pt 2): 397-403
• 38 Lesselroth BJ, Felder RS, Adams SM. , et al. Design and implementation of a medication reconciliation kiosk: the Automated Patient History Intake Device (APHID). J Am Med Inform Assoc 2009; 16 (03) 300-304
• 39 Lesselroth B, Adams K, Tallett S. , et al. Usability evaluation of a Medication Reconciliation and Allergy Review (MRAR) kiosk: a methodological approach for analyzing user interactions. Stud Health Technol Inform 2015; 218: 40598
• 40 Lesselroth BJ, Adams K, Tallett S. , et al. Design of admission medication reconciliation technology: a human factors approach to requirements and prototyping. HERD 2013; 6 (03) 30-48
• 41 Lesselroth BJ, Adams K, Simon SR, Boockvar KS, Kaboli PJ. , eds. Innovative approaches to medication reconciliation within the Veterans' Health Administration: designing the 'Magic Pill'. Proceedings of the American Medical Informatics Association Annual Symposium; 2014 ; Washington, DC: American Medical Informatics Association
• 42 Department of Veterans Affairs Veterans Health Administration. VHA Directive 2011–012: Medication Reconciliation. In: Administration VH. , ed. Washington, DC: Veterans Health Administration; 2011: 7
• 43 Lesselroth B, Adams S, Felder R. , et al. Using consumer-based kiosk technology to improve and standardize medication reconciliation in a specialty care setting. Jt Comm J Qual Patient Saf 2009; 35 (05) 264-270
• 44 Schnipper JL, Gandhi TK, Wald JS. , et al. Design and implementation of a web-based patient portal linked to an electronic health record designed to improve medication safety: the Patient Gateway medications module. Inform Prim Care 2008; 16 (02) 147-155
• 45 Lasky T, Kogut S, Campbell S, Risica PM. Computer kiosks to deliver medication information in the pharmacy. J Consum Health Internet 2011; 15 (04) 347-360
• 46 Heyworth L, Paquin AM, Clark J. , et al. Engaging patients in medication reconciliation via a patient portal following hospital discharge. J Am Med Inform Assoc 2014; 21 (e1): e157-e162
• 47 Schnipper JL, Gandhi TK, Wald JS. , et al. Effects of an online personal health record on medication accuracy and safety: a cluster-randomized trial. J Am Med Inform Assoc 2012; 19 (05) 728-734
• 48 Cohen AN, Chinman MJ, Hamilton AB, Whelan F, Young AS. Using patient-facing kiosks to support quality improvement at mental health clinics. Med Care 2013; 51 (03) (Suppl. 01) S13-S20
• 49 Zeng-Treitler Q, Perri S, Nakamura C. , et al. Evaluation of a pictograph enhancement system for patient instruction: a recall study. J Am Med Inform Assoc 2014; 21 (06) 1026-1031
• 50 Hornick TR, Higgins PA, Stollings C, Wetzel L, Barzilai K, Wolpaw D. Initial evaluation of a computer-based medication management tool in a geriatric clinic. Am J Geriatr Pharmacother 2006; 4 (01) 62-69
• 51 Kimmel SE, Lewis JD, Jaskowiak J, Kishel L, Hennessy S. Enhancement of medication recall using medication pictures and lists in telephone interviews. Pharmacoepidemiol Drug Saf 2003; 12 (01) 1-8
• 52 Carayon P, Schoofs Hundt A, Karsh B-T. , et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 (Suppl. 01) i50-i58
• 53 Lesselroth BJ, Dorr DA, Adams K. , et al. Medication review software to improve the accuracy of outpatient medication histories: protocol for a randomized controlled trial. Hum Factors Ergon Manuf 2012; 22 (01) 72-86 Epub October 14, 2011
• 54 Donabedian A. The quality of medical care. Science 1978; 200 (4344): 856-864
• 55 Medina J. Brain Rules. Seattle, WA: Pear Press; 2008
• 56 Paivio A, Rogers TB, Smythe P. Why are pictures easier to recall than words?. Psychon Sci 1968; 11 (04) 137-138
• 57 Lidwell W, Holden K, Butler J. Universal Principles of Design. Beverly, MA: Rockport Publishers; 2003
• 58 Houts PS, Doak CC, Doak LG, Loscalzo MJ. The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Patient Educ Couns 2006; 61 (02) 173-190
• 59 Raynor DK, Dickinson D. Key principles to guide development of consumer medicine information--content analysis of information design texts. Ann Pharmacother 2009; 43 (04) 700-706
• 60 Machtinger EL, Wang F, Chen L-L, Rodriguez M, Wu S, Schillinger D. A visual medication schedule to improve anticoagulation control: a randomized, controlled trial. Jt Comm J Qual Patient Saf 2007; 33 (10) 625-635
• 61 Knapp P, Raynor DK, Jebar AH, Price SJ. Interpretation of medication pictograms by adults in the UK. Ann Pharmacother 2005; 39 (7-8): 1227-1233
• 62 Coulter A, Vessey M, McPherson K, Crossley B. The ability of women to recall their oral contraceptive histories. Contraception 1986; 33 (02) 127-137
• 63 Lambert BL, Chang K-Y, Lin S-J. Effect of orthographic and phonological similarity on false recognition of drug names. Soc Sci Med 2001; 52 (12) 1843-1857
• 64 Lambert BL, Lin S-J, Chang K-Y, Gandhi SK. Similarity as a risk factor in drug-name confusion errors: the look-alike (orthographic) and sound-alike (phonetic) model. Med Care 1999; 37 (12) 1214-1225
• 65 Rataboli PV, Garg A. Confusing brand names: nightmare of medical profession. J Postgrad Med 2005; 51 (01) 13-16
• 66 US Food and Drug Administration. National Drug Code Directory. Silver Spring, MD: U.S. Department of Health and Human Services; 2011 [updated February 4, 2016]. Available at: https://www.accessdata.fda.gov/scripts/cder/ndc/ . Accessed January 24, 2018
• 67 Kaboli PJ, McClimon BJ, Hoth AB, Barnett MJ. Assessing the accuracy of computerized medication histories. Am J Manag Care 2004; 10 (11 Pt 2): 872-877
• 68 Strunk LB, Matson AW, Steinke D. Impact of a pharmacist on medication reconciliation on a patient admission to a Veterans Affairs Medical Center. Hosp Pharm 2008; 43 (08) 643-649
• 69 Jaeschke R, Guyatt GH, Sackett DL. Users' guides to the medical literature. III. How to use an article about a diagnostic test. B. What are the results and will they help me in caring for my patients? The Evidence-Based Medicine Working Group. JAMA 1994; 271 (09) 703-707
• 70 Schulz KF, Altman DG, Moher D. ; CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 2010; 152 (11) 726-732
• 71 Meyer GJ. Guidelines for reporting information in studies of diagnostic test accuracy: the STARD initiative. J Pers Assess 2003; 81 (03) 191-193
• 72 Chan KS, Fowles JB, Weiner JP. Review: electronic health records and the reliability and validity of quality measures: a review of the literature. Med Care Res Rev 2010; 67 (05) 503-527
• 73 Linsky A, Simon SR. Medication discrepancies in integrated electronic health records. BMJ Qual Saf 2013; 22 (02) 103-109
• 74 Varkey P, Cunningham J, Bisping DS. Improving medication reconciliation in the outpatient setting. Jt Comm J Qual Patient Saf 2007; 33 (05) 286-292
• 75 Gleason KM, Groszek JM, Sullivan C, Rooney D, Barnard C, Noskin GA. Reconciliation of discrepancies in medication histories and admission orders of newly hospitalized patients. Am J Health Syst Pharm 2004; 61 (16) 1689-1695
• 76 Kostas T, Paquin AM, Zimmerman K, Simone M, Skarf LM, Rudolph JL. Characterizing medication discrepancies among older adults during transitions of care: a systematic review focusing on discrepancy synonyms, data sources and classification terms. Aging Health 2013; 9 (05) 497-508
• 77 Stewart M. The validity of an interview to assess a patient's drug taking. Am J Prev Med 1987; 3 (02) 95-100
• 78 Boockvar KS, Liu S, Goldstein N, Nebeker J, Siu A, Fried T. Prescribing discrepancies likely to cause adverse drug events after patient transfer. Qual Saf Health Care 2009; 18 (01) 32-36
• 79 Badowski SA, Rosenbloom D, Dawson PH. Clinical importance of pharmacist-obtained medication histories using a validated questionnaire. Am J Hosp Pharm 1984; 41 (04) 731-732
• 80 Pippins JR, Gandhi TK, Hamann C. , et al. Classifying and predicting errors of inpatient medication reconciliation. J Gen Intern Med 2008; 23 (09) 1414-1422
• 81 Smith JD, Coleman EA, Min S-J. A new tool for identifying discrepancies in postacute medications for community-dwelling older adults. Am J Geriatr Pharmacother 2004; 2 (02) 141-147
• 82 Wong JD, Bajcar JM, Wong GG. , et al. Medication reconciliation at hospital discharge: evaluating discrepancies. Ann Pharmacother 2008; 42 (10) 1373-1379
• 83 Fung KW, Kayaalp M, Callaghan F, McDonald CJ. Comparison of electronic pharmacy prescription records with manually collected medication histories in an emergency department. Ann Emerg Med 2013; 62 (03) 205-211
• 84 Kushniruk A, Santos SL, Pourakis G, Nebeker JR, Boockvar KS. , Eds. Cognitive analysis of a medication reconciliation tool: applying laboratory and naturalistic approaches to system evaluation. ITCH 2011. ; February 2011; Victoria, Canada
• 85 Wolff CM, Nowacki AS, Yeh J-Y, Hickner JM. A randomized controlled trial of two interventions to improve medication reconciliation. J Am Board Fam Med 2014; 27 (03) 347-355
• 86 Walsh EK, Cussen K. “Take ten minutes”: a dedicated ten minute medication review reduces polypharmacy in the elderly. Ir Med J 2010; 103 (08) 236-238
• 87 Vermeire E, Hearnshaw H, Van Royen P, Denekens J. Patient adherence to treatment: three decades of research. A comprehensive review. J Clin Pharm Ther 2001; 26 (05) 331-342
• 88 Moisan J, Gaudet M, Grégoire J-P, Bouchard R. Non-compliance with drug treatment and reading difficulties with regard to prescription labelling among seniors. Gerontology 2002; 48 (01) 44-51
• 89 Stroupe KT, Smith BM, Hogan TP. , et al. Medication acquisition across systems of care and patient-provider communication among older veterans. Am J Health Syst Pharm 2013; 70 (09) 804-813
• 90 Cornu P, Steurbaut S, Leysen T. , et al. Effect of medication reconciliation at hospital admission on medication discrepancies during hospitalization and at discharge for geriatric patients. Ann Pharmacother 2012; 46 (04) 484-494
• 91 Andersen SE, Pedersen AB, Bach KF. Medication history on internal medicine wards: assessment of extra information collected from second drug interviews and GP lists. Pharmacoepidemiol Drug Saf 2003; 12 (06) 491-498
• 92 Coleman JJ, Hodson J, Thomas SK, Brooks HL, Ferner RE. Temporal and other factors that influence the time doctors take to prescribe using an electronic prescribing system. J Am Med Inform Assoc 2015; 22 (01) 206-212
• 93 Stephenson BJ, Rowe BH, Haynes RB, Macharia WM, Leon G. The rational clinical examination. Is this patient taking the treatment as prescribed?. JAMA 1993; 269 (21) 2779-2781
• 94 Bourgeois FT, Porter SC, Valim C, Jackson T, Cook EF, Mandl KD. The value of patient self-report for disease surveillance. J Am Med Inform Assoc 2007; 14 (06) 765-771
• 95 Fowler Jr F. Survey Research Methods. Los Angeles, CA: Sage; 2014: 172
• 96 Mekonnen AB, McLachlan AJ, Brien JA. Pharmacy-led medication reconciliation programmes at hospital transitions: a systematic review and meta-analysis. J Clin Pharm Ther 2016; 41 (02) 128-144
• 97 Lesselroth BJ, Eisenhauer W, Adams SM. , et al. Simulation modeling of a check-in and medication reconciliation ambulatory clinic kiosk. J Healthc Eng 2011; 2 (02) 197-222
• 98 Grant RW, Devita NG, Singer DE, Meigs JB. Improving adherence and reducing medication discrepancies in patients with diabetes. Ann Pharmacother 2003; 37 (7-8): 962-969
• 99 Stock R, Scott J, Gurtel S. Using an electronic prescribing system to ensure accurate medication lists in a large multidisciplinary medical group. Jt Comm J Qual Patient Saf 2009; 35 (05) 271-277
• 100 Marien S, Krug B, Spinewine A. Electronic tools to support medication reconciliation: a systematic review. J Am Med Inform Assoc 2017; 24 (01) 227-240
• 101 McCarthy L, Su XW, Crown N. , et al. Medication reconciliation interventions in ambulatory care: A scoping review. Am J Health Syst Pharm 2016; 73 (22) 1845-1857
• 102 Weingart SN, Cleary A, Seger A. , et al. Medication reconciliation in ambulatory oncology. Jt Comm J Qual Patient Saf 2007; 33 (12) 750-757
• 103 Kausler D. Experimental Psychology, Cognition, and Human Aging. New York, NY: Springer-Verlag; 1991
• 104 Grindrod K, Burns CM. , eds. Medication reconciliation: more than just a check. 2015 International Symposium on Human Factors and Ergonomics in Health Care: Improving the Outcomes; April 26–29, 2015 ; Baltimore, MD: Human Factors and Ergonomics Society
• 105 Bhupatiraju RT, Gorman PN. , eds. “Doing the yellows”–analysis of medication review processes by different clinicians in long term care. American Medical Informatics Association Annual Symposium, November 6, 2008; Washington, DC: American Medical Informatics Association; 2008
• 106 Jylhä V, Saranto K. Electronic documentation in medication reconciliation - a challenge for health care professionals. Appl Nurs Res 2008; 21 (04) 237-239
• 107 White CM, Schoettker PJ, Conway PH. , et al. Utilising improvement science methods to optimise medication reconciliation. BMJ Qual Saf 2011; 20 (04) 372-380
• 108 Plaisant C, Wu J, Hettinger AZ, Powsner S, Shneiderman B. Novel user interface design for medication reconciliation: an evaluation of Twinlist. J Am Med Inform Assoc 2015; 22 (02) 340-349
• 109 Nelson SJ, Zeng K, Kilbourne J, Powell T, Moore R. Normalized names for clinical drugs: RxNorm at 6 years. J Am Med Inform Assoc 2011; 18 (04) 441-448
• 110 Liu S, Wei M, Moore R, Ganesan V, Nelson SJ. RxNorm: prescription for electronic drug information exchange. IT Prof 2005; 7 (05) 17-23
• 111 Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated systematic review with a focus on meaningful use. Ann Intern Med 2014; 160 (01) 48-54
• 112 Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010; 363 (06) 501-504
• 113 Nazi KM, Turvey CL, Klein DM, Hogan TP, Woods SSVA. VA OpenNotes: exploring the experiences of early patient adopters with access to clinical notes. J Am Med Inform Assoc 2015; 22 (02) 380-389
• 114 Nazi KM. Veterans' voices: use of the American Customer Satisfaction Index (ACSI) Survey to identify My HealtheVet personal health record users' characteristics, needs, and preferences. J Am Med Inform Assoc 2010; 17 (02) 203-211
• 115 Bailey SC, Belter LT, Pandit AU, Carpenter DM, Carlos E, Wolf MS. The availability, functionality, and quality of mobile applications supporting medication self-management. J Am Med Inform Assoc 2014; 21 (03) 542-546
• 116 Goldzweig CL, Orshansky G, Paige NM. , et al. Electronic patient portals: evidence on health outcomes, satisfaction, efficiency, and attitudes: a systematic review. Ann Intern Med 2013; 159 (10) 677-687
• 117 Lyles C, Tieu L, Schillinger D, Ratanawongsa N, Sarkar U. , eds. Early experiences with meaningful use and online portal implementation among providers/staff and patients/caregivers in a safety net healthcare system. American Medical Informatics Association Fall Symposium 2015 . San Francisco, CA: American Medical Informatics Association; 2015
• 118 Rappaport DI, Collins B, Koster A. , et al. Implementing medication reconciliation in outpatient pediatrics. Pediatrics 2011; 128 (06) e1600-e1607
• 119 Ratanawongsa N, Karter AJ, Parker MM. , et al. Communication and medication refill adherence: the Diabetes Study of Northern California. JAMA Intern Med 2013; 173 (03) 210-218
• 120 Tang PC, Newcomb C. Informing patients: a guide for providing patient health information. J Am Med Inform Assoc 1998; 5 (06) 563-570
• 121 Checchi KD, Huybrechts KF, Avorn J, Kesselheim AS. Electronic medication packaging devices and medication adherence: a systematic review. JAMA 2014; 312 (12) 1237-1247
• 122 Istepanian R, Jovanov E, Zhang YT. Introduction to the special section on M-Health: beyond seamless mobility and global wireless health-care connectivity. IEEE Trans Inf Technol Biomed 2004; 8 (04) 405-414
• 123 McGavock H, Britten N, Weinman T. A Review of the Literature on Drug Adherence. London: Royal Pharmaceutical Society of Great Britain; 1996: 1-55
• 124 Redd T, Doberne J, Lattin D. , et al , eds. Variability in electronic health record usage and perceptions among specialty vs. primary care physicians. American Medical Informatics Association Fall Symposium 2015 . San Francisco, CA; 2015
• 125 McDowell SE, Ferner HS, Ferner RE. The pathophysiology of medication errors: how and where they arise. Br J Clin Pharmacol 2009; 67 (06) 605-613