Conceptual Design, Implementation, and Evaluation of Generic and Standard-Compliant Data Transfer into Electronic Health Records

Abstract

Objectives The objective of this study is the conceptual design, implementation and evaluation of a system for generic, standard-compliant data transfer into electronic health records (EHRs). This includes patient data from clinical research and medical care that has been semantically annotated and enhanced with metadata. The implementation is based on the single-source approach. Technical and clinical feasibilities, as well as cost-benefit efficiency, were investigated in everyday clinical practice.

Methods Münster University Hospital is a tertiary care hospital with 1,457 beds and 10,823 staff who treated 548,110 patients in 2018. Single-source metadata architecture transformation (SMA:T) was implemented as an extension to the EHR system. This architecture uses Model Driven Software Development (MDSD) to generate documentation forms according to the Clinical Data Interchange Standards Consortium (CDISC) operational data model (ODM). Clinical data are stored in ODM format in the EHR system database. Documentation forms are based on Google's Material Design Standard. SMA:T was used at a total of five clinics and one administrative department in the period from March 1, 2018 until March 31, 2019 in everyday clinical practice.

Results The technical and clinical feasibility of SMA:T was demonstrated in the course of the study. Seventeen documentation forms including 373 data items were created with SMA:T. Those were created for 2,484 patients by 283 users in everyday clinical practice. A total of 121 documentation forms were examined retrospectively. The Constructive cost model (COCOMO II) was used to calculate cost and time savings. The form development mean time was reduced by 83.4% from 3,357 to 557 hours. Average costs per form went down from EUR 953 to 158.

Conclusion Automated generic transfer of standard-compliant data and metadata into EHRs is technically and clinically feasible, cost efficient, and a useful method to establish comprehensive and semantically annotated clinical documentation. Savings of time and personnel resources are possible.

Keywords

hospital information systems - electronic health records - medical informatics - operational data model - single-source metadata architecture transformation

Background and Significance

Scientific Background

Documentation in clinical research and medical care is a resource-intensive and complex process. Heterogeneous standards and specifications are used in electronic health records (EHRs) and electronic data capture (EDC) systems.[1] [2] Redundant data storage results from limited functionality of existing EHR systems, which frequently do not yet fulfill regulatory requirements for clinical studies.[3] Due to different standards and terminology systems, the existing interoperability in the health care sector is limited.[4] [5] [6] [7] Medical professionals document one patient's data in two systems, the EHR and EDC systems. Medical informatics professionals need to develop documentation forms in both systems with very similar content. Redundant and complex steps can be reduced significantly using the single-source approach and increased data quality can be achieved.[8] [9] However, development effort and documentation processes are complex and demanding. The broad range of pathologies requires a large number of different data elements to be recorded, with more than 13,000 diagnoses (International Classification of Diseases and Related Health Problems—German Modification [ICD-10 GM]) only for reimbursement purposes (even much more for detailed clinical diagnoses). There is a great need for adapted specialist documentation.[10] The single-source concept and generic creation of standard-compliant documentation forms offer an opportunity to reduce this workload. Efficient development, reusability of data, and avoiding redundant input are positive aspects of this approach which allows for more rapid documentation.[11] [12] The reusability of structured EHR data are required in clinical and translational research[13] [14] [15] [16] [17] [18] and improves safety, quality, and efficiency in the health care sector.[19] [20] Reusability of EHR data in research is currently limited due to a lack of standardization in EHR systems.[21]

Objectives

Objective of the Study

The objective of this study is the conceptual design, implementation, and evaluation of a generic, standard-compliant data transfer into EHRs. This transfer includes patient data from clinical research and medical care that has been semantically annotated and enhanced with metadata. Implementation shall be based on the single-source approach.[22] The study aims to answer the following three research questions:

1. Is generic, standard-compliant data transfer into EHRs technically feasible?

2. Will clinical users accept generic, standard-compliant documentation forms?

3. Is generic data transfer is cost efficient regarding the development of documentation forms?

Setting

Münster University Hospital in Germany is a tertiary care hospital with 1,457 beds and 10,823 staff who treated 548,110 patients (inbound and outbound) in 2018.[23] Generic data transfer was used in everyday clinical practice at five clinics: psychiatry and psychotherapy,[24] general pediatrics,[25] general and visceral surgery,[26] phoniatrics and pediatric audiology,[27] and neurology,[28] as well as at the administrative department for palliative medicine.[29] Validation was performed by 137 doctors and 146 specialists from the health care sector in the period from March 1, 2018 to March 31, 2019.

System Details

The EHR system ORBIS by Agfa HealthCare[30] is used at Münster University Hospital in more than 40 clinics and is the market leader in Germany with 780 installations. A large amount of documentation is needed in everyday clinical practice. More than 1,500 documentation forms are in active use at Münster University Hospital. Design and maintenance of clinical documentation are performed with a proprietary form builder, which is similar to a What You See Is What You Get (WYSIWYG) editor with limited functionalities. It is a resource-intensive process with approximately 31 new developments and 954 change requests per year.[31] The EHR system has a 3,462 GB oracle database, 7,612 users, and 1,789 user sessions per day (status at July 2019). The standard version of ORBIS does not support standardized form metadata and clinical data or annotated datasets.

Requirements

Requirement Engineering

Requirement engineering was applied to assess clinical documentation needs regarding research and routine care. Unstructured interviews with clinical users were performed and evaluated. Clinical requirements must be implemented more efficiently. Standards for data processing in external systems were identified. Development of clinical documentation forms is an iterative process, therefore a model driven approach[32] should be applied to generate executable applications. Scalability of the development process had to be achieved. To address regulatory requirements, standards from clinical research needed to be applied for data transfer into EHRs. Clinical documentation forms were to be created generically on the basis of a standard-compliant structure. These should be stored in the database of the EHR system. Accessibility to all processes and data in the EHR system had to be achieved. The underlying architecture had to be based on the five scenarios of clinical trial documentation according to the electronic source data interchange (eSDI)[33] group: source at site, e-source system provider, single source concept, and extraction and investigator verification, as well as direct extraction from EHRs. To achieve efficiency, a single-source strategy should be applied, by which the data are entered only once into the EHR system. This approach eliminates data transcription and ensures interoperability between EHR and EDC systems. To satisfy regulatory requirements of Food and Drug Administration (FDA) and European Medicines Agency (EMA), Clinical Data Interchange Standards Consortium (CDISC) operational data models (ODMs/define-XML)[34] [35] [36] [37] [38] should be used. Standardized metadata should be used to take quality assurance processes into account and to minimized EHR bias.[39] [40] [41] Local data protection and IT (information and technology) security rules of the University Hospital must be met.

Solution Requirements

Single-source metadata architecture transformation (SMA:T) was derived from the named requirements. CDISC ODM (version 1.3.2) was used as a flexible standard for exchange and archiving of metadata within the framework of clinical studies.[42] Data transfer into the EHR system was performed via a communication server. ODM files were transported automatically to the database of the EHR system with health level 7 (HL7) messages.[43] HL7 version 2.5 and message type ORU^R01 were used. The EHR database contains complete and comprehensive ODM data structures for internal and external processing. NextGen Connect[44] was used as a communication server. Documentation forms of SMA:T were based on Google's Material Design Standard[45] to make it adaptable to differing corporate designs using templates. Plausibility and completeness of form data had be validated by clinical users.

Methods

Analysis of Technical and Clinical Feasibility

Technical feasibility was demonstrated by the implementation of SMA:T. The clinical feasibility was accessed through a prospective analysis of clinical documentation forms which were created with SMA:T. A total of 137 doctors and 146 health care sector specialists were clinical users of the system. The IT business division at Münster University Hospital is maintaining the EHR system.[46] Evaluation began on March 1, 2018 when SMA:T went live in the EHR system of Münster University Hospital. From this point onward, incoming requests from the clinics were developed with SMA:T. Documentation frequency was observed and evaluated until March 31, 2019. The following evaluation criteria were employed:

1. Measurement of data completeness in created documentation forms

2. Monitoring of system stability

IBM SPSS Statistics version 25[47] was used for descriptive data analysis. Adobe Photoshop version 11.0[48] and Microsoft Visio version 16.0.4849.1000[49] were used to depict the workflow.

Cost-Benefit Analysis

The development time for clinical documentation forms was analyzed retrospectively. Development effort was determined in person months (PMs). Among 383, 121 custom documentation forms were examined for this purpose. Due to the retrospective setting, a source code analysis was used, since other indicators, such as time measurements or detailed information about EHR specifications, were not available; therefore use-case points method or similar could not be applied. The constructive cost model (COCOMO II)[50] was used to estimate costs and effort (details in Eq. 1). Parameters were selected in accordance with the recommendations of COCOMO II: A = 2.94, E = 1.1788, effort adjusted factor (EAF) = 0.34 (conventional creation method) and EAF = 0.47 (SMA:T). The delivered source instructions (DSI) were determined by the number of source lines of code (SLOC) in the documentation form. Evaluation criteria were established using the following four parameters:

1. Monetary analysis and comparison of the methods to create clinical documentation forms.

2. Software development time.

3. Time to develop a documentation form.

4. Determination of the break-even point for economic efficiency based on the number of data elements and documentation forms.

Formulas

PM = A * (DSI/1000)^E * EAF

Eq. 1 The COCOMO II was used to estimate costs and effort. The effort was determined in PMs. The parameters were selected in accordance with the recommendations of OCOMO II: A = 2.94, E = 1.1788, EAF = 0.34 (conventional creation method) and EAF = 0.47 (SMA:T). The constant A denotes the multiplicative effects on the effort for projects. The scale factor E captures the relative economy (or diseconomy) of scale encountered for software projects. The EAF is used to capture characteristics of the software development process that affect the effort to complete the project. Delivered source instructions (DSI) were determined by lines of code in the documentation form.

Personnel costs were calculated (provided by the human resource [HR] department). The following three groups of persons were involved.

1. Students.

2. IT professionals.

3. Medical informatics professionals.

The first group of people includes students who are paid in Germany in accordance with the “mini-job” pay scale. The costs of IT professionals were determined using the average of the T11–T12 pay category; for medical informatics professionals the T13 pay category was applied.[51] The summary of personnel costs can be found in [Table 1]. It was thus possible to determine effort and costs for form creation. This effort was determined in minutes. Costs were calculated based on personnel costs (hourly rate HR). One PM corresponds to 152 hours in accordance with the COCOMO II procedure.

Table 1 Personnel costs by pay category for students, IT professionals, and medical informatics professionals in euros per hour

Job title	Hourly rate (€)
Student	9.23
IT professional	24.94
Medical informatics professional	34.38

Abbreviation: IT, information and technology.

Results

System Architecture

SMA:T was implemented as an extension of the EHR system. It provides documentation templates directly based on EHR functionality with data storage within the EHR database. The software architecture uses Model Driven Software Development (MDSD) to generate executable applications. A metamodel defines modules and rules by which the applications are generated. Components, data types, connectors, and interfaces are specified through additions to the ODM structure. Code generators are used to create the application logic, and factories[52] are applied to create the front end. ODM structures are used as input parameters. An ODM file is to be understood as a model in this context. A comparison with rules in the metamodel is made prior to generation to avoid overhead. There are currently two implementation variants for use in everyday clinical practice (details in [Fig. 1]).

Fig. 1 Unified modeling language (UML) activity diagram of the generic creation of an application by SMA:T. Clinical documentation forms are created on the basis of an ODM structure. Model driven software development (MDSD) is used within the software architecture. Modular applications are created by a fully automated process and embedded applications are created manually. ODM, operational data model; SMA:T, single-source metadata architecture transformation.

Modular Application (Fully Automated)

A modular application is created once by a medical informatics professional. It comprises a generator template and a link to a dictionary. This dictionary is stored in the database of the EHR system and can contain any number of ODM files. These files are provided automatically by the communication server.

Embedded Application (Customized)

An embedded application is created individually and links to an ODM file. The file is manually stored in the database of the EHR system.

All data fields are subject to semantic encoding with Unified Medical Language System (UMLS) codes[53] which are important for the reuse of EHR data.[54] UMLS is a metathesaurus and is suitable for efficient and semiautomatic coding of data elements like eligibility criteria.[55] Semantic coding was implemented according to the existing concepts of the university hospital[56] [57] to facilitate data integration with external data sources in a study context. The workflow of SMA:T is shown in [Fig. 2].

Fig. 2 Unified modeling language (UML) sequence diagram of the SMA:T workflow. In process steps 1–8, the metadata are created by a Medical Informatics Professionals using an integrated development environment of his choice. The automatic data transfer is initiated by the communication server. This fetches the created ODM files from a file share and transfers them to the database of the EHR system. In process steps 9–10, clinical users are assigned access rights to these forms. In process steps 11–15, user can enter and store patient data with these forms. EHR, electronic health record; ODM, operational data model; SMA:T, single-source metadata architecture transformation.

System Implementation

Implementation of the architecture covers frontend and backend. Agile methods were used on the Project Life Cycle and Development Cycle.[58] Additional content is available via Figshare.[59]

Frontend

The frontend was designed in accordance with Google's Material Design Standard ([Figs. 3] and [4]). Different usability standards for the display of templates[60] [61] [62] were analyzed and taken into account. For the most part, items were designed in accordance with the guidelines for usable web form design by Seckler et al.[63] The arrangement is vertical in a single-column layout. Titles, labels, and user interface elements follow the material design standard. Items have context-related modular tool bars.

Fig. 3 Documentation form regarding palliative care created with SMA:T based on the material design standard from Google. SMA:T, single-source metadata architecture transformation.

Fig. 4 Documentation form regarding psychiatry created with SMA:T based on the material design standard from Google. SMA:T, single-source metadata architecture transformation.

Backend

The backend structure comprises import and export of metadata, as well as clinical data. Data storage takes place within the EHR database in a separate scheme. Import and export of metadata and data are controlled via HL7 channels in the communication server. Metadata transfer applies a complete ODM structure which is stored in the EHR database. External data can be transferred from third party applications, smart devices, and wearables. These data are imported to the EHR database in ODM format (ODM clinical data). Each dataset is patient-related and clearly references an existing ODM file from the metadata area. Portability is supported through the use of templates for modules and architectural components.

Study Findings and Outcome Data

Technical and Clinical Feasibility

SMA:T was used at six clinics/administrative departments. Seventeen documentation forms including 373 data items were implemented for this purpose. A total of 2,836 instances were created by the users at Münster University Hospital and 2,484 cases were opened for 2,484 patients by 283 users ([Table 2]). Twelve professional groups worked with SMA:T. These included the following: case manager, medical assistant, medical controller, medical technician, nurse, physician, psychologist/therapist, revenue management, scientist, secretary, social services, and study assistant. Due to the complete integration of SMA:T into the EHR system, all data protection aspects, user rights/roles and security features of the EHR system are available for SMA:T. Standardized data transfer from the communication server into the EHR system was completed without error. It was possible to display all items (n = 373) from ODM structures in full using the generic workflow. An 8-bit UCS transformation format (UTF-8)-based characters[64] in the ODM structure were displayed correctly in the frontend and backend of the EHR system. Clinical data (n = 2,484 patients) was fully stored in the EHR database. ODM-based data export worked correctly without any errors. Automatic generation of documentation forms was accepted in routine clinical use.

Table 2 The number of SMA:T form instances created per clinic/institute at Münster University Hospital

Clinic/department	No. of instances	No. of cases
General pediatrics	9	5
General, visceral and transplant surgery	92	88
Neurology	48	47
Palliative care	422
Phoniatrics	1,643	1,618
Psychiatry	622	304

Abbreviations: SMA:T, single-source metadata architecture transformation.

Note: Number of patient cases is also shown in table.

Cost-Benefit Analysis

SMA:T reduced the effort of implementing forms compared with the conventional process. The need to create a label and an input field for each item, including their individual positioning and configuration, was reduced to creating one ODM file. Development time for SMA:T was 85,956 minutes (9.4 PMs); this corresponds to EUR 49,246 personnel costs for a Medical Informatics Professional. Results are shown in [Table 3]. The monetary comparison between SMA:T and conventional EHR documentation forms was based on personnel costs. To determine the time effort, 121 documentation forms were studied retrospectively over a period of four years (2014–2017). The average time effort for one form with the conventional method is 1,664 minutes, and with SMA:T, it is 276 minutes. Development costs were calculated based on personnel costs. Cost savings of EUR 96,247 were achieved with SMA:T compared with the conventional method. Extensible markup language (XML)[65]-based creation of documentation forms does not require a Medical Informatics Professional and can be performed by an IT professional or a documentation assistant. Development time in accordance with COCOMO II was 22.1 PM conventionally and only 3.7 PM with SMA:T. This demonstrates the economic benefit of the SMA:T technology. Results are shown in [Table 4]. The time effort for the prospective implementation of 17 documentation forms was 2,419 (SMA:T) and 14,623 minutes (conventional). Implementation times and costs were determined in analogy to set-up costs. Results are shown in [Table 5]. The break-even point was calculated based on set-up costs of SMA:T and form costs. Set-up costs were EUR 49,246. The break-even point is thus reached at 52 documentation forms. Commercial implementation services for SMA:T are available via [e-mail].

Table 3 Development time and costs of the SMA:T software architecture within the ORBIS EHR system of Agfa HealthCare GmbH

	SMA:T
Forms	74
SLOC	13,711
Items	1,110
Hours	1,432
Person month	9.4
Costs (€)	49,246

Abbreviations: EHR, electronic health record; SLOC, source lines of code; SMA:T, single-source metadata architecture transformation.

Table 4 Comparison of development time and costs of SMA:T and conventionally developed documentation forms (retrospective analysis)

	Conventional	SMA:T	Cost/time savings
Average time effort for one form (min)	1,664	276	1,388
Person month	22.1	3.7	18.4
Development costs (€)	115,406	19,159	96,247

Abbreviation: SMA:T, single-source metadata architecture transformation.

Table 5 Comparison of development time and costs of SMA:T and conventionally developed documentation forms (prospective analysis)

Forms	Items	Time (min) SMA:T	Time (min) conventional	Costs (€) SMA:T	Costs (€) conventional
ACH QM Bogen MS	36	250	1,510	143	865
DMNZ Mini Nutritional Assessment MS	37	258	1,559	148	893
KIALL Untersuchungsbefund Neonatologie MS	21	132	799	76	458
NEURO Off-Label-Use MS	26	170	1,029	97	590
PALL Palliativmedizin Verlaufsbogen MS	12	68	413	39	237
PHON Patientenbogen MS	24	155	936	89	536
PHON Vormerkung MS	20	125	755	72	433
PSYCH BFI-2-S MS	37	258	1,559	148	893
PSYCH BFI-2-XS MS	22	140	845	80	484
PSYCH Beck Depression Inventory MS	25	163	982	93	563
PSYCH Childhood Trauma Questionnaire MS	36	250	1,510	143	865
PSYCH Familienanamnese MS	16	96	580	55	332
PSYCH Koerperliche Erkrankungen MS	7	36	219	21	125
PSYCH NARQ-S MS	11	61	373	35	214
PSYCH Somatische Symptome MS	16	96	580	55	332
PSYCH Soziodemographischer Bogen MS	7	36	219	21	125
PSYCH Symptomverlauf MS	20	125	755	72	433

Abbreviation: SMA:T, single-source metadata architecture transformation.

Unexpected Observations

There were two unexpected observations in the course of the evaluation phase. First, there was a delay in the use of new documentation forms. Internal communication processes in the clinics delayed effective use and increased the time window from provision until use. Furthermore, the high level of acceptance for new documentation forms led to a rapid increase in further orders from the clinics.

Discussion

Answers to the Study Questions

The aim of the study was to evaluate a generic, standard-compliant data transfer into EHRs, regarding clinical acceptance and cost-benefit efficiency. The study showed that this generic, standard-compliant data transfer is technically feasible. Furthermore, it demonstrated that manual work steps during the creation of documentation forms, which are prone to errors, could be reduced. Standardization improved data quality in the hospital information system. It was possible to reduce development times. There was acceptance from different clinics and institutes at the University Hospital. Those documentation forms are used in an interdisciplinary manner in routine patient care. An impressive cost efficiency could be demonstrated.

Strengths and Weaknesses of the Study

Strengths

This study demonstrated a clear reduction in the development time of clinical documentation forms. Generic creation of documentation forms (which is not covered by the current HL7 standards) is suitable for efficient extensions and updates of EHR systems in everyday clinical practice. The MDSD approach was used to prevent redundancy, homonymy, and misclassification, as well as combining modularization, problem separation, and reuse with efficient implementation. Standardized data transfer was performed with ODM format. ODM is the format that is mandated by regulatory authorities (FDA) for data and metadata in clinical studies.

Weaknesses

This is a single-site study that is a limitation on scalability. Neither workflow mechanisms nor individual input templates are currently supported by SMA:T. Our evaluation concentrates on technical and clinical feasibility. However, data from EHR systems need to be interpreted carefully because of potential EHR bias.[39] [40] [41] Before reusing EHR data for research purposes, appropriate quality management processes need to be established. Further evaluation is necessary to assess the sustainable benefit in everyday clinical practice.

Results in Relation to Other Studies

One objective of this study is to foster reuse of clinical data in a research context by flexible adaptation of documentation to avoid redundant data entry. Ethier[66] used dual modeling based on the Clinical Research Information Model (CRIM) and the Clinical Data Integration Model (CDIM) of meaning. CDISC ODM was used for metadata and data transfer. To extract clinical data, the interoperability framework separates domain information from heterogeneous data sources where necessary to achieve structured data exchange, as specified by the models CRIM and CDIM. Matsumura et al developed an EDC system that works with an electronic medical record (EMR) system and automatically converts and sends all necessary data from EMR to an electronic case report form (e-CRF).[67] [68] In a study by El Fadly et al,[69] a new implementation of “extraction and researcher verification” was performed according to the e-source data interchange document. A semantic interoperability framework was defined to support reuse of clinical data. A mediator was implemented for the transformation of CDISC ODM into a proprietary XML for different EHR solutions. The substitutable medical applications, reusable technology (SMART) on fast health care interoperability resources (FHIR) standard applies a different approach.[70] This is an open standard for the development of health care applications and is based on the HL7 FHIR standard. SMART on FHIR allows for a data exchange using OAuth2 and OpenID between client and server. FHIR resources are used for data exchange. An FHIR server is needed. However, many EHR systems currently do not use an FHIR server. SMART on FHIR is compatible with SMA:T. All those previous studies require either a second system or data transformation to be able to use clinical data in a research context. In this study, the existing EHR system was expanded according to the single-source approach: clinical documentation is stored in the database of the EHR system on the basis of the ODM. This enables collection and extraction of assessments based on the ODM standard directly from the patient's medical records as an ODM file. Clinical data are provided in the ODM format for efficient further processing in EDC systems. For research purposes, patient data needs to be provided in pseudonymized form. Our results have shown that such a system is technically feasible, accepted in the clinical setting and cost efficient. Therefore it offers advantages both for routine care and clinical research.

Generalizability of the Study

The conversion of EHRs to semantically annotated, structured patient records is a decisive step for the implementation of machine learning, artificial intelligence, and clinical decision support solutions. This work has shown that generic data transfer into EHRs is feasible and economical. The implementation of semantically annotated documentation within hospital information systems can provide benefits for future medicine, like the reuse of data. Synchronous distribution of documentation is possible with comprehensive implementation of SMA:T in hospitals. This is of great relevance for both quality assurance and clinical research regarding data collection. The metalevel of patient records is interesting in principle for all clinics, hospitals and university hospitals. Automated and intelligent solutions in the health care sector could be established comprehensively and quickly with this kind of software architecture.

Future Work

SMA:T should be evaluated at different sites, because many data collection efforts need to be implemented in several locations simultaneously. The prospective implementation of SMA:T in EHR systems with other software architectures is needed for successful multisite tests. SMA:T was developed in an ORBIS site. EHR systems from other software vendors must re-implement SMA:T in their respective software environment to process CDISC ODM files. A software blueprint can be made available. It is expected that standard-compliant, real-time data transfer from other data sources like smart devices, wearables, and artificial intelligence applications into clinical documentation can potentially improve patient care and reduce the workload of clinical staff. The SMA:T architecture provides a standard-based metadata and data transfer of EHR systems with external systems and thereby can contribute to such integrated EHR systems.

Conclusion

Automated and generic transfer of standard-compliant data and metadata to EHRs is technically and clinically feasible, cost effective, and useful to establish comprehensive and semantically annotated clinical documentation. Savings of both time and personnel resources are possible.

Clinical Relevance Statement

The SMA:T architecture provides cost-effective, fast-to-implement standardized data transfer/collection into EHR systems. Translational areas of medicine can be established in clinics in this way. Multisite data transfer and exchange of uniform standardized clinical documentation forms are possible.

Multiple Choice Questions

1. What was a model-driven approach used for?

   • Execution of applications.

   • Creation of executable applications.

   • Reduction of personnel costs.

   • Generation of electronic clinical research forms

   Correct Answer: The correct answer is option b. A model-driven approach was used for the efficient creation of executable applications. Each application represents a clinical documentation form in a standardized format.

2. Which method was used to determine the cost-benefit analysis?

   • Stopwatch.

   • Function point analysis.

   • Constructive cost analysis.

   • Constructive cost model (COCOMO II).

   Correct Answer: The correct answer is option d. The constructive cost model (COCOMO II) was used for cost and effort estimation. The development effort could thus be determined in person-months.

3. What is SMA:T?

   • EHR system.

   • Standard format for data transfer.

   • Single-source metadata architecture transformation for the extension of EHR systems.

   • EDC system.

   Correct Answer: The correct answer is option c. The single-source metadata architecture transformation (SMA:T) was implemented as an extension to the EHR system.

4. Which format was used for the data transfer?

   • CDISC operational data model (Version 1.3.2).

   • Fast healthcare interoperability resources.

   • Health level 7.

   • CDISC operational data model (Version 1.3.1).

   Correct Answer: The correct answer is option a. The CDISC operational data model (Version 1.3.2) was used for data transfer. This format was used to meet regulatory requirements of FDA and EMA.

Conflict of Interest

None declared.

Acknowledgments

General support from the team of the Business Unit IT, University Hospital Münster is acknowledged.

Protection of Human and Animal Subjects

This manuscript contains no patient data; therefore, it is not subject to human subject research approval.

• References

• 1 Tapuria A, Bruland P, Delaney B, Kalra D, Curcin V. Comparison and transformation between CDISC ODM and EN13606 EHR standards in connecting EHR data with clinical trial research data. Digit Health 2018; 4: 2055207618777676
• 2 Maldonado JA, Marcos M, Fernández-Breis JT. , et al. A platform for exploration into chaining of web services for clinical data transformation and reasoning. AMIA Annu Symp Proc 2017; 2016 (04) 854-863
• 3 Herzberg S, Rahbar K, Stegger L, Schäfers M, Dugas M. Concept and Implementation of a Single Source Information System in Nuclear Medicine for Myocardial Scintigraphy (SPECT-CT data). Appl Clin Inform 2010; 1 (01) 50-67
• 4 Laleci GB, Yuksel M, Dogac A. Providing semantic interoperability between clinical care and clinical research domains. IEEE J Biomed Health Inform 2013; 17 (02) 356-369
• 5 Weng C, Appelbaum P, Hripcsak G. , et al. Using EHRs to integrate research with patient care: promises and challenges. J Am Med Inform Assoc 2012; 19 (05) 684-687
• 6 Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research applications and clinical care. Nat Rev Genet 2012; 13 (06) 395-405
• 7 Kropf S, Chalopin C, Lindner D, Denecke K. Domain modeling and application development of an archetype- and XML-based EHRS. Practical experiences and lessons learnt. Appl Clin Inform 2017; 8 (02) 660-679
• 8 Bruland P, Forster C, Breil B, Ständer S, Dugas M, Fritz F. Does single-source create an added value? Evaluating the impact of introducing x4T into the clinical routine on workflow modifications, data quality and cost-benefit. Int J Med Inform 2014; 83 (12) 915-928
• 9 Forster C, Bruland P, Lechtenbörger J, Breil B, Vossen G. Connecting clinical care and research: Single-source with x4T – Process design, architecture, and use cases. Inf Tecnol 2015; 57 (01) 1211
• 10 Optum. ICD-10-CM Clinical Documentation Improvement Desk Reference. Optuminsight Inc.; 2018
• 11 Prokosch HU, Ganslandt T. Perspectives for medical informatics. Reusing the electronic medical record for clinical research. Methods Inf Med 2009; 48 (01) 38-44
• 12 Baumann LA, Baker J, Elshaug AG. The impact of electronic health record systems on clinical documentation times: a systematic review. Health Policy 2018; 122 (08) 827-836
• 13 Wang Y, Wang L, Rastegar-Mojarad M. , et al. Clinical information extraction applications: A literature review. J Biomed Inform 2018; 77: 34-49
• 14 Shenvi EC, Meeker D, Boxwala AA. Understanding data requirements of retrospective studies. Int J Med Inform 2015; 84 (01) 76-84
• 15 Köpcke F, Trinczek B, Majeed RW. , et al. Evaluation of data completeness in the electronic health record for the purpose of patient recruitment into clinical trials: a retrospective analysis of element presence. BMC Med Inform Decis Mak 2013; 13 (01) 37
• 16 Shwarz-Boeger U, Kiechle M, Paepke S. , et al. EHR and EDC Integration in Reality. Available at: http://www.appliedclinicaltrialsonline.com/ehr-and-edc-integration-reality . Accessed October 13, 2019
• 17 Scholte RA, Opmeer BC, Ploem MC. [From record keeping to scientific research: obstacles and opportunities for research with electronic health records] (in Dutch). Ned Tijdschr Geneeskd 2017; 161: D1738
• 18 Rosenbloom ST, Carroll RJ, Warner JL, Matheny ME, Denny JC. Representing knowledge consistently across health systems. Yearb Med Inform 2017; 26 (01) 139-147
• 19 van Velthoven MH, Mastellos N, Majeed A, O'Donoghue J, Car J. Feasibility of extracting data from electronic medical records for research: an international comparative study. BMC Med Inform Decis Mak 2016; 16 (01) 90
• 20 Watterson JL, Rodriguez HP, Aguilera A, Shortell SM. Ease of use of electronic health records and relational coordination among primary care team members. Health Care Manage Rev 2018; DOI: 10.1097/HMR.0000000000000222.
• 21 Bonomi L, Jiang X. Linking temporal medical records using non-protected health information data. Stat Methods Med Res 2018; 27 (11) 3304-3324
• 22 Dugas M, Breil B, Thiemann V, Lechtenbörger J, Vossen G. Single source information systems to connect patient care and clinical research. Stud Health Technol Inform 2009; 150: 61-65
• 23 Universitätsklinikum Münster UKM. Available at: https://www.ukm.de/ . Accessed October 13, 2019
• 24 UKM. Klinik für Psychiatrie und Psychotherapie der Universitätsklinik Münster. Available at: https://www.ukm.de/index.php?id=psychiatrie_uebersicht . Accessed October 13, 2019
• 25 UKM. Klinik für Kinder- und Jugendmedizin (Allgemeine Pädiatrie) der Universitätsklinik Münster. ukm.de. Available at: https://www.ukm.de/index.php?id=paed_uebersicht . Accessed October 13, 2019
• 26 UKM. Klinik für Allgemein- und Viszeralchirurgie der Universitätsklinik Münster. Available at: https://www.ukm.de/index.php?id=chiurgie_uebersicht . Accessed October 13, 2019
• 27 UKM. Klinik für Phoniatrie und Pädaudiologie der Universitätsklinik Münster. Available at: https://www.ukm.de/index.php?id=phoniatrie_uebersicht . Accessed October 13, 2019
• 28 UKM. Klinik für Neurologie mit Institut für Translationale Neurologie. Available at: https://www.ukm.de/index.php?id=neurologie_uebersicht . Accessed October 13, 2019
• 29 Palliativmedizin der Universitätsklinik Münster UKM. Available at: https://www.ukm.de/index.php?id=palliativmedizin . Accessed October 13, 2019
• 30 AGFA. ORBIS, das Krankenhaus-Informationssystem von Agfa HealthCare. Available at: https://global.agfahealthcare.com/dach/krankenhaus-informationssystem/ . Accessed October 13, 2019
• 31 Blitz R DM. 62. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e. V. (GMDS). Chronos–Ein xml-basiertes User-Interface-Framework zur effizienten Erstellung klinischer Dokumentationsbögen. 2017: Doi:10.3205/17gmds110
• 32 Stahl T, Efftinge S, Haase A, Völter M. Modellgetriebene Softwareentwicklung. dpunkt.verlag; 2012
• 33 CDISC. eSource Data Interchange Document. Available at: https://www.cdisc.org/esdi-document . Accessed October 13, 2019
• 34 FDA< CDER, CBER. Study data technical conformance guide. Available at: https://www.fda.gov/media/122913/download . Accessed October 13, 2019
• 35 CDISC. ODM-XML. Available at: https://www.cdisc.org/standards/data-exchange/odm . Accessed October 13, 2019
• 36 CDISC. Clinical Data Interchange Standards Consortium. Available at: http://www.cdisc.org . Accessed October 13, 2019
• 37 CDISC. Glossary, Abbreviation and Acronyms. Available at: https://www.cdisc.org/standards/glossary . Accessed October 13, 2019
• 38 CDISC. Define-XML. Available at: https://www.cdisc.org/standards/data-exchange/define-xml . Accessed October 13, 2019.
• 39 Lynch KE, Deppen SA, DuVall SL. , et al. Incrementally transforming electronic medical records into the observational medical outcomes partnership common data model: a multidimensional quality assurance approach. Appl Clin Inform 2019; 10 (05) 794-803
• 40 Agniel D, Kohane IS, Weber GM. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ 2018; 361: k1479
• 41 Verheij RA, Curcin V, Delaney BC, McGilchrist MM. Possible Sources of Bias in Primary Care Electronic Health Record Data Use and Reuse. J Med Internet Res 2018; 20 (05) e185
• 42 Soto-Rey I, Neuhaus P, Bruland P. , et al. Standardising the development of ODM converters: the ODMToolBox. Stud Health Technol Inform 2018; 247: 231-235
• 43 Health Level Seven International (HL7). Available at: http://www.hl7.org/ . Accessed October 13, 2019
• 44 NextGen. NextGen. Available at: https://www.nextgen.com/ . Accessed October 13, 2019
• 45 Google. Design guidance and code. Available at: https://material.io/ . Accessed October 13, 2019
• 46 UKM. Universitätsklinikum Münster Geschäftsbereich IT.Available at: https://www.ukm.de/index.php?id=gb-it . Accessed October 13, 2019
• 47 IBM. IBM SPSS Statistics. Available at: https://www.ibm.com/analytics/spss-statistics-software . Accessed October 13, 2019
• 48 Adobe. Adobe Photoshop.Available at: https://www.adobe.com/de/products/photoshop.html . Accessed October 13, 2019
• 49 Microsoft. Microsoft Visio. Available at: https://products.office.com/de-de/visio/flowchart-software?tab=tabs-1 . Accessed October 13, 2019
• 50 University of Southern California. COCOMO II. Available at: http://sunset.usc.edu/csse/affiliate/private/COCOMOII_2000/COCOMOII-040600/modelman.pdf . Accessed October 13, 2019
• 51 Fern-Uni-Hagen. Fern-Uni-Hagen (Ed.): TV-L Arbeitgeberberechnung. fernuni-hagen.de. Available at: http://www.fernuni-hagen.de/arbeiten/personalthemen/bezahlung/index.shtml . Accessed October 13, 2019
• 52 Gamma E, Helm R, Johnson R, Vlissides J. Design Patterns: Entwurfsmuster als Elemente wiederverwendbarer objektorientierter Software. MITP-Verlags GmbH & Co. KG;2015
• 53 Medicine USNLO. Unified Medical Language System (UMLS). Available at: https://www.nlm.nih.gov/research/umls/ . Accessed October 13, 2019
• 54 Williams R, Kontopantelis E, Buchan I, Peek N. Clinical code set engineering for reusing EHR data for research: A review. J Biomed Inform 2017; 70: 1-13
• 55 Luo Z, Yetisgen-Yildiz M, Weng C. Dynamic categorization of clinical research eligibility criteria by hierarchical clustering. J Biomed Inform 2011; 44 (06) 927-935
• 56 Varghese J, Dugas M. Frequency analysis of medical concepts in clinical trials and their coverage in MeSH and SNOMED-CT. Methods Inf Med 2015; 54 (01) 83-92
• 57 Varghese J, Sandmann S, Dugas M. Web-based information infrastructure increases the interrater reliability of medical coders: quasi-experimental study. J Med Internet Res 2018; 20 (10) e274
• 58 Kannan V, Basit MA, Youngblood JE. , et al. Agile co-development for clinical adoption and adaptation of Innovative Technologies. Health Innov Point Care Conf 2018; 56-59 ; Doi: 10.1109/HIC.2017.8227583
• 59 Blitz R. Single-source Metadata ARchitecture Transformation - Supplement. Doi: 10.6084/m9.figshare.12011592
• 60 Bargas-Avila JA, Brenzikofer O, Roth SP, User AT. 2010 Simple but crucial user interfaces in the world wide web: introducing 20 guidelines for usable web form design. Intechopencom. DOI: 10.5772/9500. Available at: https://www.intechopen.com/books/user-interfaces/simple-but-crucial-user-interfaces-in-the-world-wide-web-introducing-20-guidelines-for-usable-web-fo . Accessed April 9, 2020
• 61 Idrus Z, Razak N. , ICCEA NTAA, 2010 Using Three Layer Model (TLM) in web form design: WeFDeC checklist development. IEEE. Published in: 2010 Second International Conference on Computer Engineering and Applications. Doi: 10.1109/ICCEA.2010.81
• 62 United States. Dept. of Health, Human Services, & United States. General Services Administration. Research-based Web Design & Usability Guidelines. U.S. Department of Health and Human Services; 2006
• 63 Seckler M, Heinz S, Bargas-Avila JA. , et al. Designing Usable Web Forms: Empirical Evaluation of Web Form Improvement Guidelines. Available at: https://dl.acm.org/doi/abs/10.1145/2556288.2557265 . Accessed April 9, 2020
• 64 FY. UTF-8, a transformation format of ISO 10646. Available at: https://tools.ietf.org/pdf/rfc3629.pdf . Accessed October 13, 2019
• 65 World Wide Web Consortium. Extensible Markup Language (XML). Available at: http://www.w3.org/TR/2006/REC-xml11-20060816/ . Accessed October 13, 2019
• 66 Ethier J-F, Curcin V, McGilchrist MM. , et al. eSource for clinical trials: Implementation and evaluation of a standards-based approach in a real world trial. Int J Med Inform 2017; 106: 17-24
• 67 Matsumura Y, Hattori A, Manabe S. , et al. Interconnection of electronic medical record with clinical data management system by CDISC ODM. Stud Health Technol Inform 2014; 205: 868-872
• 68 Matsumura Y, Hattori A, Manabe S. , et al. Case Report Form Reporter: A Key Component for the Integration of Electronic Medical Records and the Electronic Data Capture System. Stud Health Technol Inform 2017; 245: 516-520
• 69 El Fadly A, Lucas N, Rance B, Verplancke P, Lastic P-Y, Daniel C. The REUSE project: EHR as single datasource for biomedical research. Stud Health Technol Inform 2010; 160 (Pt 2): 1324-1328
• 70 Program BCHCHI, Informatics THMSDFB. Substitutable Medical Applications and Reusable Technologies (SMART). Available at: https://smarthealthit.org . Accessed October 13, 2019