Summary
Objectives: The aim of this study is to objectify user critique rendering it usable for quality
assurance. Based on formative and summative evaluation results we strive to promote
software improvements; in our case, the physician discharge letter composition process
at the Department of Dermatology, University Hospital Erlangen, Germany.
Methods : We developed a novel six-step approach to objectify user critique: 1) acquisition
of user critique using subjectivist methods, 2) creation of a workflow model, 3) definition
of hypothesis and indicators, 4) measuring of indicators, 5) analyzing results, 6)
optimization of the system regarding both subjectivist and objectivist evaluation
results. In particular, we derived indicators and workflows directly from user critique/narratives.
The identified indicators were mapped onto work-flow activities, creating a link between
user critique and the evaluated system.
Results: Users criticized a new discharge letter system as “too slow” and “too labor-intensive”
in comparison with the previously used system. In a stepwise approach we collected
subjective user critique, derived a comprehensive process model including deviations
and deduced a set of five indicators for objectivist evaluation: processing time,
system-related waiting time, number of mouse clicks, number of keyboard inputs, and
throughput time. About 3500 measurements have been performed to compare the work-flow-steps
of both systems, regarding 20 discharge letters.
Although the difference of the mean total processing time between both systems was
statistically insignificant (2011.7 s vs. 1971.5 s; p = 0.457), we detected a significant
difference in waiting times (101.8 s vs. 37.2 s; p < 0.001) and number of user interactions
(77 vs. 69; p < 0.001) in favor of the old system, thus objectifying user critique.
Conclusions: Our six-step approach enables objectification of user critique, resulting in objective
values for continuous quality assurance. To our knowledge no previous study in medical
informatics mapped user critique onto workflow steps. Subjectivist analysis prompted
us to use the indicator system-related waiting time for the objectivist study, which
was rarely done before. We consider combining subjectivist and objectivist methods
as a key point of our approach. Future work will concentrate on automated measurement
of indicators.
Keywords Hospital information systems - patient discharge - evaluation studies - attitude to
computers - quality assurance - health care
