Appl Clin Inform 2016; 07(02): 368-379
DOI: 10.4338/ACI-2015-11-RA-0161
Research Article - H3IT Special Topic
Schattauer GmbH

Using Electronic Case Summaries to Elicit Multi-Disciplinary Expert Knowledge about Referrals to Post-Acute Care

Kathryn H. Bowles
1   University of Pennsylvania School of Nursing, Philadelphia, PA
3   Visiting Nurse Service of New York
Sarah Ratcliffe
2   University of Pennsylvania Perelman School of Medicine, Philadelphia, PA
Sheryl Potashnik
1   University of Pennsylvania School of Nursing, Philadelphia, PA
Maxim Topaz
1   University of Pennsylvania School of Nursing, Philadelphia, PA
John Holmes
2   University of Pennsylvania Perelman School of Medicine, Philadelphia, PA
Nai-Wei Shih
1   University of Pennsylvania School of Nursing, Philadelphia, PA
Mary D. Naylor
1   University of Pennsylvania School of Nursing, Philadelphia, PA
› Author Affiliations
The authors wish to thank the National Institute of Nursing Research of the National Institutes of Health under Award Number R01-NR007674 and the experts who participated in this study for supporting the research reported in this publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We also thank the nursing and information technology staff at the six hospitals who assisted us in collecting the electronic data for the study and MI Digital Agency for their technical expertise in building the website and data management.
Further Information

Publication History

received: 23 November 2015

accepted: 28 February 2016

Publication Date:
16 December 2017 (online)



Eliciting knowledge from geographically dispersed experts given their time and scheduling constraints, while maintaining anonymity among them, presents multiple challenges.


Describe an innovative, Internet based method to acquire knowledge from experts regarding patients who need post-acute referrals. Compare, 1) the percentage of patients referred by experts to percentage of patients actually referred by hospital clinicians, 2) experts’ referral decisions by disciplines and geographic regions, and 3) most common factors deemed important by discipline.


De-identified case studies, developed from electronic health records (EHR), contained a comprehensive description of 1,496 acute care inpatients. In teams of three, physicians, nurses, social workers, and physical therapists reviewed case studies and assessed the need for post-acute care referrals; Delphi rounds followed when team members did not agree. Generalized estimating equations (GEEs) compared experts’ decisions by discipline, region of the country and to the decisions made by study hospital clinicians, adjusting for the repeated observations from each expert and case. Frequencies determined the most common case characteristics chosen as important by the experts.


The experts recommended referral for 80% of the cases; the actual discharge disposition of the patients showed referrals for 67%. Experts from the Northeast and Midwest referred 5% more cases than experts from the West. Physicians and nurses referred patients at similar rates while both referred more often than social workers. Differences by discipline were seen in the factors identified as important to the decision.


The method for eliciting expert knowledge enabled national dispersed expert clinicians to anonymously review case summaries and make decisions about post-acute care referrals. Having time and a comprehensive case summary may have assisted experts to identify more patients in need of post-acute care than the hospital clinicians. The methodology produced the data needed to develop an expert decision support system for discharge planning.

Citation: Bowles KH, Ratcliffe S, Potashnik S, Topaz M, Holmes J, Shih N-W, Naylor MD. Using electronic case summaries to elicit multi-disciplinary expert knowledge about referrals to post-acute care.

  • References

  • 1 Bowles KH, Foust JB, Naylor MD. Hospital discharge referral decision making: a multidisciplinary perspective. Appl Nurs Res 2003; Aug; 16 (03) 134-143.
  • 2 Milton NR. Knowledge acquisition in practice: a step-by-step guide. Springer; 2007
  • 3 Musen MA, Fagan LM, Combs DM, Shortliffe EH. Use of a domain model to drive an interactive knowledge-editing tool. Int J Human Comput Stud 1999; 51 (02) 479-495.
  • 4 Dexter F, Wachtel RE, Epstein RH. Event-based knowledge elicitation of operating room management decision-making using scenarios adapted from information systems data. BMC Med Inform Decis Mak 2011; 11: 02 6947–11–2.
  • 5 Gibert K, García-Alonso C, Salvador-Carulla L. Integrating clinicians, knowledge and data: Expert-based cooperative analysis in healthcare decision support. Health Res Policy and Systems 2010; 8.
  • 6 Groznik V, Guid M, Sadikov A, Mozina M, Georgiev D, Kragelj V, Ribaric S, Pirtosek Z, Bratko I. Elicitation of neurological knowledge with argument-based machine learning. Artif Intell Med 2013; 57 (02) 133-144.
  • 7 Keune H, Gutleb AC, Zimmer KE, Ravnum S, Yang A, Bartonova A, Krayer von Krauss M, Ropstad E, Eriksen GS, Saunders M, Magnati B, Forsberg B. We’re only in it for the knowledge? A problem solving turn in environment and health expert elicitation. Environ Health 2012; 11 (Suppl. 01) S3 069X-11-S1-S3.
  • 8 Gavrilova T, Andreeva T. Knowledge elicitation techniques in a knowledge management context. J Knowledge Management 2012; 16 (04) 523-537.
  • 9 Ziebell D, Fiore SM, Becerra-Fernandez I. Knowledge Management Revisited. IEEE Intelligent Systems 2008; 23 (03) 84-88.
  • 10 Okafor EC, Osuagwu CC. The underlying issues in knowledge elicitation. Interdisc J of Information, Knowledge, and Management 2006; 01: 95-108.
  • 11 Cooke NJ. Knowledge Elicitation. In: Durso FT, Nickerson RS, Schvaneveldt RW, Dumais ST, Lindsay DS, Chi MTH. editors. Handbook of Applied Cognition. First ed.. Chichester, New York: Wiley; 1999: 479-501.
  • 12 Bech-Larsen T, Nielsen NA. A comparison of five elicitation techniques for elicitation of attributes of low involvement products. J Econ Psych 1999; 20: 315-341.
  • 13 Orem DE. Nursing: Concepts of Practice. 5th ed.. St. Louis, MO: Mosby; 1995
  • 14 Bowles KH, Holmes JH, Naylor MD, Liberatore M, Nydick R. Expert consensus for discharge referral decisions using online delphi. AMIA Annu Symp Proc 2003; 106-9.
  • 15 Bowles K, Potashnik S, Ratcliffe S, Rosenberg M, Shih N, Topaz M, Holmes J, Naylor M. Conducting research using the electronic health record across multi-hospital systems: semantic harmonization implications for administrators. J Nurs Admin 2013; 43 (06) 355-360.
  • 16 Ai MY, Li K, Lin DJK. Balanced incomplete Latin square designs. J Statistical Planning and Inference 2013; 143: 1575-1582.
  • 17 Rowe G, Wright G. The Delphi technique as a forecasting tool: issues and analysis. Int J Forecasting 1999; 15 (04) 353-375.
  • 18 Liang KY, Zeger S. Longitudinal data analysis using generalized linear models. Biometrics 1986; 73: 13-22.
  • 19 Zeger S, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics 1986; 42: 121-130.
  • 20 Walden G, Schwartz A. Walden, Schwartz introduce bipartisan legislation to ensure seniors and disabled can access home health services. [Internet]. Washington, DC: U.S. Representative Greg Walden; 2015. [cited January 18, 2016]. Available from:
  • 21 Maramba PJ, Richards S, Larrabee JH. Discharge Planning Process, Applying a Model for Evidence-Based Practice. J Nurs Care Qual 2004; 19 (02) 123-129.
  • 22 Holland DE, Bowles KH. Standardized discharge planning assessments: impact on patient outcomes. J Nurs Care Qual 2012; 27 (03) 200-208.