Download PDF
Methods Inf Med 2011; 50(01): 23-35
DOI: 10.3414/ME09-01-0052
Original Articles
Schattauer GmbH

Objectifying User Critique

A Means of Continuous Quality Assurance for Physician Discharge Letter Composition
M. Oschem
1   Medical Center for Information and Communication Technology, University Hospital Erlangen, Erlangen, Germany
,
V. Mahler
2   Department of Dermatology, University Hospital Erlangen, Erlangen, Germany
,
H. U. Prokosch
3   Chair of Medical Informatics, Department of Medical Informatics, Biometrics and Epidemiology, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany
› Author Affiliations
Further Information

Publication History

received: 16 June 2009

accepted: 22 January 2009

Publication Date:
18 January 2018 (online)

    Permissions and Reprints

Summary

Objectives: The aim of this study is to objectify user critique rendering it usable for quality assurance. Based on formative and summative evaluation results we strive to promote software improvements; in our case, the physician discharge letter composition process at the Department of Dermatology, University Hospital Erlangen, Germany.

Methods: We developed a novel six-step approach to objectify user critique: 1) acquisition of user critique using subjectivist methods, 2) creation of a workflow model, 3) definition of hypothesis and indicators, 4) measuring of indicators, 5) analyzing results, 6) optimization of the system regarding both subjectivist and objectivist evaluation results. In particular, we derived indicators and workflows directly from user critique/narratives. The identified indicators were mapped onto work-flow activities, creating a link between user critique and the evaluated system.

Results: Users criticized a new discharge letter system as “too slow” and “too labor-intensive” in comparison with the previously used system. In a stepwise approach we collected subjective user critique, derived a comprehensive process model including deviations and deduced a set of five indicators for objectivist evaluation: processing time, system-related waiting time, number of mouse clicks, number of keyboard inputs, and throughput time. About 3500 measurements have been performed to compare the work-flow-steps of both systems, regarding 20 discharge letters.

Although the difference of the mean total processing time between both systems was statistically insignificant (2011.7 s vs. 1971.5 s; p = 0.457), we detected a significant difference in waiting times (101.8 s vs. 37.2 s; p < 0.001) and number of user interactions (77 vs. 69; p < 0.001) in favor of the old system, thus objectifying user critique.

Conclusions: Our six-step approach enables objectification of user critique, resulting in objective values for continuous quality assurance. To our knowledge no previous study in medical informatics mapped user critique onto workflow steps. Subjectivist analysis prompted us to use the indicator system-related waiting time for the objectivist study, which was rarely done before. We consider combining subjectivist and objectivist methods as a key point of our approach. Future work will concentrate on automated measurement of indicators.

Keywords

Hospital information systems - patient discharge - evaluation studies - attitude to computers - quality assurance - health care
 

  • References

  • 1 Modler H. Qualitätsmanagement: Der Arztbrief. Göttingen:: Universität Göttingen, Bereich Human-medizin, Medizinische Informatik; 2005. (updated 2005, cited 28.8.2007). Available from: http://www.mi.med.uni-goettingen.de
    Google Scholar
  • 2 Semler SC. Automatische Arztbriefschreibung – was sollte sie können, wie weit darf sie gehen?. In: Wehrs H. editor. Der Computer-Führer für Ärzte. 9th ed 2001
    Google Scholar
  • 3 Spießl H, Cording C. Kurz, strukturiert und rasch übermittelt: Der “optimale” Arztbrief. Deutsche Medizinische Wochenschrift 2001; 126: 184-187.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 4 Waegemann CP. Where is eHealth going?. 2008 (updated 2008; cited 21.10.2008). Available from: http://www.medrecinst.com
    PubMedGoogle Scholar
  • 5 Collen MF. Fifty years in medical informatics. Methods Inf Med 2006; 45 (01) 174-179.
    PubMedGoogle Scholar
  • 6 Klar R. Selected impressions on the beginning of the electronic medical record and patient information. Methods Inf Med 2004; 43 (05) 537-542.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 7 Peterson HE. From punched cards to computerized patient records: a personal journey. Methods Inf Med 2006; 45 (01) 180-186.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 8 Roberts MS, Dreese EM, Hurley N, Zullo N, Peterson M. Blending administrative and clinical needs: the development of a referring physician database and automatic referral letter. Proc Annu Symp Comput Appl Med Care 1991 pp 559-563.
    PubMedGoogle Scholar
  • 9 Aarts J, Berg M. Same systems, different outcomes – comparing the implementation of computerized physician order entry in two Dutch hospitals. Methods Inf Med 2006; 45 (01) 53-61.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 10 Aarts J, Doorewaard H, Berg M. Understanding implementation: the case of a computerized physician order entry system in a large Dutch university medical center. J Am Med Inform Assoc 2004; 11 (03) 207-216.
    CrossrefPubMedGoogle Scholar
  • 11 Southon G, Sauer C, Dampney K. Lessons from a failed information systems initiative: issues for complex organisations. Int J Med Inform 1999; 55 (01) 33-46.
    CrossrefPubMedGoogle Scholar
  • 12 Ash JS, Anderson JG, Gorman PN, Zielstorff RD, Norcross N, Pettit J. et al. Managing change: analysis of a hypothetical case. J Am Med Inform Assoc 2000; 7 (02) 125-134.
    CrossrefPubMedGoogle Scholar
  • 13 Larndorfer R. Evaluation des klinischen Informationssystems an der Universitätsklinik für Unfallchirurgie und Sporttraumatologie Innsbruck. Inns-bruck:: Private Universität für Gesundheitswissenschaften, Medizinische Informatik und Technik; 2005
    Google Scholar
  • 14 Lechleitner G, Pfeiffer KP, Wilhelmy I, Ball M. Cerner Millennium: the Innsbruck experience. Methods Inf Med 2003; 42 (01) 8-15.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 15 Bürkle T, Ammenwerth E, Prokosch HU, Dudeck J. Evaluation of clinical information systems. What can be evaluated and what cannot?. Journal of Evaluation in Clinical Practice 2001; 7 (04) 373-385.
    CrossrefPubMedGoogle Scholar
  • 16 Bürkle T. Quality of Healthcare: The Role of Informatics. Stuttgart 2003 pp 235-239.
    PubMedGoogle Scholar
  • 17 Archbold RA, Laji K, Suliman A, Ranjadayalan K, Hemingway H, Timmis AD. Evaluation of a computer-generated discharge summary for patients with acute coronary syndromes. Br J Gen Pract 1998; 48 429 1163-1164.
    PubMedGoogle Scholar
  • 18 van Walraven C, Laupacis A, Seth R, Wells G. Dictated versus database-generated discharge summaries: a randomized clinical trial. Cmaj 1999; 160 (03) 319-326.
    PubMedGoogle Scholar
  • 19 Lissauer T, Paterson CM, Simons A, Beard RW. Evaluation of computer generated neonatal discharge summaries. Arch Dis Child 1991; 66 4 Spec No 433-436.
    CrossrefPubMedGoogle Scholar
  • 20 Bolton P. A quality assurance activity to improve discharge communication with general practice. J Qual Clin Pract 2001; 21 (03) 69-70.
    CrossrefPubMedGoogle Scholar
  • 21 Moorman PW, Branger PJ, van der Kam WJ, van der Lei J. Electronic messaging between primary and secondary care: a four-year case report. J Am Med Inform Assoc 2001; 8 (04) 372-378.
    CrossrefPubMedGoogle Scholar
  • 22 Callen JL, Alderton M, McIntosh J. Evaluation of electronic discharge summaries: a comparison of documentation in electronic and handwritten discharge summaries. Int J Med Inform 2008; 77 (09) 613-620.
    CrossrefPubMedGoogle Scholar
  • 23 Walker CA. Dictated versus hand-written accident and emergency discharge documents. Emerg Med J 2007; 24 (01) 67.
    PubMedGoogle Scholar
  • 24 Roth-Isigkeit A, Harder S. Die Entlassmedikation im Arztbrief. Medizinische Klinik 2005; 100 (02) 87-93.
    CrossrefPubMedGoogle Scholar
  • 25 Lieb K, Trieschmann S, Buhl C, Vielhaber K, Mehraein S, Härter MM. Verkürzung der Übermittlungsdauer von Arztbriefen stationär behandelter Patienten in der Psychiatrie und Psychotherapie. Nervenarzt 2007; 78: 322-327.
    CrossrefPubMedGoogle Scholar
  • 26 Paterson JM, Allega RL. Improving communication between hospital and community physicians. Feasibility study of a handwritten, faxed hospital discharge summary. Discharge Summary Study Group. Can Fam Physician 1999; 45: 2893-2899.
    PubMedGoogle Scholar
  • 27 O’Leary KJ, Liebovitz DM, Feinglass J, Liss DT, Evans DB, Kulkarni N. et al. Creating a better discharge summary: improvement in quality and timeliness using an electronic discharge summary. J Hosp Med 2009; 4 (04) 219-225.
    CrossrefPubMedGoogle Scholar
  • 28 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-112.
    PubMedGoogle Scholar
  • 29 Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007; 76 (01) 21-27.
    CrossrefPubMedGoogle Scholar
  • 30 Patel VL, Kushniruk AW. Understanding, navigating and communicating knowledge: issues and challenges. Methods Inf Med 1998; 37 4–5 460-470.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 31 Ammenwerth E, Kaiser F, Bürkle T, Gräber S, Herr-mann G, Wilhelmy I. Evaluation of user acceptance of data management systems in hospitals – feasibility and usability. Proceedings of the 9th European Conference on Information Technology Evaluation 2002. 2002 pp 31-38.
    PubMedGoogle Scholar
  • 32 Ammenwerth E, Mansmann U, Iller C, Eichstadter R. Factors affecting and affected by user acceptance of computer-based nursing documentation: results of a two-year study. J Am Med Inform Assoc 2003; 10 (01) 69-84.
    CrossrefPubMedGoogle Scholar
  • 33 Ammenwerth E, Eichstädter R, Haux R, Pohl U, Rebel S, Ziegler S. Systematische Evaluation von Pflegedokumentationssystemen – Studienprotokoll und erste Ergebnisse. Heidelberg:: Bericht der Abteilung Med.. Informatik 2/1999; 1999
    PubMedGoogle Scholar
  • 34 Ammenwerth E, Kaiser F. Evaluation der Benutzerzufriedenheit mit der rechnergestützten Arztbriefund Befundschreibung. Innsbruck: Private Universität für Medizinische Informatik und Technik Tirol; 2002
    Google Scholar
  • 35 Engel P. Evaluation der Einführung der elektronischen Arztbriefschreibung am UKM. Münster: Westfälische Wilhelms-Universität Münster; 2004
    Google Scholar
  • 36 Haux R, Seggewies C, Baldauf-Sobez W, Kullmann P, Reichert H, Luedecke L. et al. Soarian – workflow management applied for health care. Methods Inf Med 2003; 42 (01) 25-36.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 37 Universitätsklinikum Erlangen.. Universitätsklinikum Erlangen – Jahresbericht 2007. Erlangen; 2008
    Google Scholar
  • 38 Oschem M. Evaluation und Schaffung eines objektivierbaren Vergleichs von Methoden zur Optimie-rung der Arztbriefschreibung in der Hautklinik des Universitätsklinikums Erlangen; PhD Thesis. Erlangen:: Friedrich-Alexander-Universität ErlangenNürnberg; 2009
    Google Scholar
  • 39 Heitmann KU, Kassner A, Gehlen E, Görke HJ, Heidenreich G. Arztbrief auf Basis der HL7 Clinical Document Architecture Release 2 für das deutsche Gesundheitswesen – Implementierungsleitfaden. Berlin:: Verband der Hersteller von IT-Lösungen für das Gesundheitswesen VHitG; 2006. (updated 2006, cited 27.8.2007). Available from: http://download.vhitg.de
    Google Scholar
  • 40 Bludau HB, Wolff A, Hochlehnert AJ. Presenting XML-based medical discharge letters according to CDA. Methods Inf Med 2003; 42 (05) 552-556.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 41 Ammenwerth E. Die Bewertung von Informationssystemen des Gesundheitswesens. Innsbruck: Private Universität für Medizinische Informatik und Technik Tirol; 2003
    Google Scholar
  • 42 Krallmann H, Frank H, Gronau N. Systemanalyse in Unternehmen. 3rd ed. München: Oldenbourg; 1999
    Google Scholar
  • 43 Friedman CP, Wyatt JC. Evaluation Methods in Biomedical Informatics. 2nd ed. Hannah KJ, Ball MJ. editors. New York: Springer; 2006
    Google Scholar
  • 44 House ER. Evaluating with validity. Beverly Hills: Sage Publications; 1980
    Google Scholar
  • 45 Burkle T, Kuch R, Prokosch HU, Dudeck J. Stepwise evaluation of information systems in an university hospital. Methods Inf Med 1999; 38 (01) 9-15.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 46 Ammenwerth E, Kaiser F, Wilhelmy I, Hofer S. Evaluation of user acceptance of information systems in health care – the value of questionnaires. Stud Health Technol Inform 2003; 95: 643-648.
    PubMedGoogle Scholar
  • 47 Ehlers F, Ammenwerth E, Eichstädter R, Haux R, Kruppa B, Parzer P. et al. Unterstützung der Organisation des Behandlungsprozesses in der Kinder-und Jugendpsychiatrie – Ist- und Schwachstellenanalyse. Universitätsklinikum Heidelberg, Abt. Medizinische Informatik; 2001
    PubMedGoogle Scholar
  • 48 Lang M, Bürkle T, Laumann S, Bauer J, Prokosch HU. Modeling the Radiology Workflow: A hands-on Comparison of established Process Modeling Languages. 51. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie 2006 Leipzig.: 2006
    Google Scholar
  • 49 Brender J. Evaluating success and failure factors in health information systems development. MIE 2006; 2006: 280-285.
    PubMedGoogle Scholar
  • 50 Prokosch HU, Engelmann U, Haas P, Handels H, Schug SH, Steyer G. et al. GMDS-Thesenpapier zur telematischen Vernetzung von Versorgungseinrichtungen im deutschen Gesundheitswesen. GMDS; 2001 (updated 2001, cited 31.8.2007). Available from: http://www.imi.med.uni-erlangen.de
    PubMedGoogle Scholar
  • 51 Heitmann KU, Schweiger R, Dudeck J. Discharge and referral data exchange using global standards – the SCIPHOX project in Germany. Int J Med Inform 2003; 70 2–3 195-203.
    CrossrefPubMedGoogle Scholar
  • 52 Hüske-Kraus D. Text generation in clinical medicine – a review. Methods Inf Med 2003; 42 (01) 51-60.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 53 Kannry J, Moore C, Karson T. Discharge communique: use of a workflow byproduct to generate an interim discharge summary. AMIA Annu Symp Proc 2003 pp 341-345.
    PubMedGoogle Scholar
  • 54 Prokosch HU. Prozessoptimierung durch moderne Krankenhaus-Informations- und Workflowsysteme. In: Eberspächer J, Picot A, Braun G. editors. EHealth: Innovations- und Wachstumsmotor für Europa – Potenziale in einem vernetzten Gesundheitsmarkt. Berlin: Springer; 2006
    Google Scholar
  • 55 Laerum H, Ellingsen G, Faxvaag A. Doctors’ use of electronic medical records systems in hospitals: cross sectional survey. Bmj 2001; 323 7325 1344-1348.
    CrossrefPubMedGoogle Scholar
  • 56 Becker J, Kugeler M, Rosemann M. Prozessmanagement – ein Leitfaden zur prozessorientierten Organisationsgestaltung. Berlin-Heidelberg: Springer; 2000
    Google Scholar
  • 57 Hirzel M, Kühn F, Gaida I. Prozessmanagement in der Praxis. Wiesbaden: GWV Fachverlage; 2008
    Google Scholar
  • 58 Friedag HR, Schmidt W. Balanced Scorecard – mehr als ein Kennzahlensystem. Freiburg: Haufe; 1999
    Google Scholar
  • 59 Müller U, Ammenwerth E, Brigl B, Gräber S, Gruetz R, Häber A. et al. Überwachung als Aufgabe des Informationsmanagements im Krankenhaus. GMS Med Inform Biom Epidemiol 2008; 2008 (4: Doc04)
    PubMedGoogle Scholar
  • 60 Ament-Rambow C. Prozesscontrolling – Möglichkeiten und Grenzen. Krankenhaus Umschau 2000; 2000 (04) 266-269.
    PubMedGoogle Scholar
  • 61 Chin J, Diehl V, Norman K. editors. Development of a Tool Measuring User Satisfaction of the Human-Computer Interface. Conference on Human Factors in Computing Systems; 1988. Washington, D. C.:
    Google Scholar
  • 62 Boy O, Ohmann C, Aust B, Eich HP, Koller M, Knode O. et al. Systematic evaluation of user satisfaction of physicians with a hospital information system – initial results. Stud Health Technol Inform 2000; 77: 518-522.
    PubMedGoogle Scholar
  • 63 White SA. Process Modeling Notations and Work-flow Patterns. IBM Corporation; 2004 (updated 2004, cited 13.2.2008). Available from: http://www.bpmn.org
    PubMedGoogle Scholar
  • 64 Hess R. The missing link to success: using a business process management system to automate and manage process improvement. J Healthc Inf Manag 2009; 23 (01) 27-33.
    PubMedGoogle Scholar
  • 65 Colsman A, Kunzmann U, Seggewies C, Mahler V. Arztbriefschreibung in der Dermatologie im Rahmen eines elektronischen PatientendatenMangements – Pilotuntersuchung unter medizinisch-ökonomischen Gesichtsspunkten. Der Hautarzt 2009 pp 1-5.
    PubMedGoogle Scholar
  • 66 Schabetsberger T, Ammenwerth E, Andreatta S, Gratl G, Haux R, Lechleitner G. et al. From a paper-based transmission of discharge summaries to electronic communication in health care regions. Int J Med Inform 2006; 75 3–4 209-215.
    CrossrefPubMedGoogle Scholar
  • 67 Stead WW. The evolution of the IAIMS: lessons for the next decade. J Am Med Inform Assoc 1997; 4 (02) S4-9. discussion S10-1.
    PubMedGoogle Scholar
  • 68 Hripcsak G. IAIMS architecture. J Am Med Inform Assoc 1997; 4 (02) S20-30.
    PubMedGoogle Scholar