Methods Inf Med 2019; 58(01): 024-030
DOI: 10.1055/s-0039-1692416
Original Article
Georg Thieme Verlag KG Stuttgart · New York

Eliciting and Exploiting Utility Coefficients in an Integrated Environment for Shared Decision-Making

Elisa Salvi
1  Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
,
Enea Parimbelli
2  MET Research Group, Telfer School of Management, University of Ottawa, Ottawa, Canada
,
Silvana Quaglini
1  Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
,
Lucia Sacchi
1  Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
› Author Affiliations
Further Information

Address for correspondence

Elisa Salvi, MS
Department of Electrical, Computer and Biomedical Engineering, University of Pavia
Via Ferrata 5, 27100, Pavia
Italy   

Publication History

10 October 2018

15 April 2019

Publication Date:
05 July 2019 (online)

 

Abstract

Background In shared decision-making, a key step is quantifying the patient's preferences in relation to all the possible outcomes of the compared clinical options. According to utility theory, this can be done by eliciting utility coefficients (UCs) from the patient. The obtained UCs are then used in decision models (e.g., decision trees). The elicitation process involves the choice of one or more elicitation methods, which is not easy for decision-makers who are unfamiliar with the theoretical framework. Moreover, to our knowledge there are no tools that integrate functionalities for UC elicitation with functionalities to run decision models that include the elicited values.

Objectives The first aim of this work is to provide decision support to the clinicians for the selection of the elicitation method. The second aim is to bridge the gap between UC elicitation and the exploitation of those UCs in shared decision-making.

Methods Based on evidence from the utility theory literature, we developed a set of production rules that recommend the optimal elicitation method(s) according to the patient's profile and health state. We then complemented this decision support tool with a functionality for quantifying and running decision trees defined through the commercial software TreeAge.

Results The result is an integrated framework for shared decision-making. Given the primary aim of this work, we focus for result evaluation on the elicitation tool. It was tested on 51 volunteers, who expressed UCs for four purposely selected health states. The insights on the collected UCs validated the rules included in the decision support system. The usability of the tool was assessed through the System Usability Scale, obtaining positive results.

Conclusion We developed an integrated environment to facilitate shared decision-making in the clinical practice. The next step is the validation of the entire framework and its use besides shared decision-making. As a matter of fact, it may also be exploited to target cost-utility analysis to a specific patient population.


#

Introduction

In a patient-centric clinical context, the care process is personalized according to the characteristics and needs of the individual patient. This can be achieved through shared decision making (SDM), a more and more increasingly adopted process in which the physician and the patient reach a consensus in solving a decision problem, by sharing information and values.[1] A key step in SDM is identifying patient preferences, so that they can be included in a model that represents a generic decision problem, to make it specific to the individual. Correctly incorporating patient preferences in SDM is fundamental, since it is known that different subjects may evaluate the same health state differently.[2] The desirability of a health state can be quantified through a utility coefficient (UC), a variable ranging from 0 to 1 that measures the quality of life (QOL) perceived by a patient in relation to such condition. A strategy for solving a clinical decision problem (i.e., selecting the optimal treatment for a specific patient) is comparing the options in terms of quality-adjusted life years (QALYs).[3] For each option, to compute QALYs for a patient, his expected survival is split in time intervals, each one (Ti ) spent in a specific condition Ci . Assuming that Ci is associated with a utility coefficient UC i , QALYs are computed as

Zoom Image

While QALYs are mostly known as the outcome measure in cost-utility analysis (CUA), where different health care programs are compared in terms of both economic affordability and health outcomes for a target population, they may be used as well for individual decision-making.[4] This article focuses on the latter, but in principle our tool may be used also for CUA. Since, according to the different perspectives, either patients or healthy individuals may be interviewed for UC elicitation,[5] [6] in the following we will simply use “interviewee” to refer to a subject from whom UCs are elicited, and “interviewer” to refer to the person leading the elicitation process.

Eliciting reliable UCs from an interviewee can be challenging, and the interviewer (ideally, a psychologist) faces two main difficulties. First, the utility theory provides multiple methods for eliciting UCs,[2] [7] among which the interviewer must select the most appropriate one. Second, a method can be administered in different ways, and it is difficult, without guidance, to follow the same procedure for all interviewees. Moreover, often the utility elicitation is not led by psychologists due to lack of resources, and UCs are elicited during medical appointments by physicians who are not familiar with the utility theory.

The utility theory provides multiple elicitation methods. Two well-known methods are the time trade-off (TTO) and standard gamble (SG),[2] which work with hypothetical scenarios. To value a suboptimal health state S, in TTO, the interviewee must choose between living his expected lifetime (T1) in the S state or to live shorter (T2 < T1) but in perfect health. T2 is varied until the interviewee is indifferent between the two options and the UC is computed as T2/T1. In SG, the interviewee must choose between living his expected life in the S state or accepting a gamble whose outcomes are complete healing or sudden death with probability p. Again, p is varied until the interviewee is indifferent between the two choices, and the UC is computed as (1–p). Thus, UCs quantify the loss (death risk in SG or life time in TTO) the interviewee is willing to accept in exchange for complete healing. A third method, the rating scale (RS),[7] requires the interviewee to rate S on a scale from 0 (= death) to 100 (= perfect health). Since RS is not based on the loss/gain theoretical background, it produces values rather than actual UCs, but it is simple to administer and is useful to rank multiple health states at the beginning of an elicitation process. The literature suggests that elicitation methods are not equally applicable to every interviewee, as they may be biased by specific characteristics of both the interviewee and the evaluated health state. Besides the interviewee's emotional status, the elicitation is influenced by the attitude of the interviewer, who may adopt different strategies to interact with the subjects.[8]

In front of all the challenges described above, health informatics may provide instruments to promote an unbiased and standardized elicitation. Over time, several computerized tools have been proposed,[9] [10] [11] [12] [13] [14] [15] but they still present significant limitations. None of them support interviewers in choosing the elicitation method; they implement a limited number of methods, most of them are not integrated with models (e.g., decision trees [DTs][16]) that exploit the obtained UCs in solving a clinical decision problem. The few[9] [13] that provide the latter functionality have been developed for a specific disease, and cannot be applied to other clinical domains.

The work presented in this article aimed at enhancing one of such tools, namely UceWeb,[14] by designing and implementing new functionalities to support users in a comprehensive shared decision analysis workflow.


#

Methods

Three innovative features are proposed (see the next three subsections). First of all, since UceWeb included only TTO, SG, and RS, and since the first two methods may be unsuitable for some interviewees, we extended the set of available elicitation methods. Second, we formalized and implemented a decision support system (DSS) to assist the interviewer in selecting the elicitation method that most suits the individual interviewee. Finally, we integrated UceWeb with a software for running DTs.

To evaluate the obtained system, we performed a pilot study, whose methods are described in the Evaluation section.

Enhanced Set of Elicitation Methods

SG and TTO show some applicability issues.[17] First, interviewees may act as zero-traders, i.e., they may refuse to trade/gamble (even hypothetically) their life-time in exchange for perfect health, independently from their perception of the examined health state.[17] This may happen, for example, when an interviewee feels responsible for children or other nonautonomous family members. In the following, interviewees having such a feeling of responsibility will be referred to as “caregivers.” Please note that this definition of “caregiver” does not mean that the interviewee is eliciting UCs on someone else's behalf. UCs elicited from caregivers show a ceiling effect toward 1.[17] Another problem concerns elicitation for temporary health states (lasting less than 1 year), since the short duration may induce interviewees to be reluctant in the trading/gambling task.[17] [18] [19] TTO is also unsuitable for valuing mild, not life-threatening, states,[17] [20] [21] since interviewees hardly trade life-time in this case. Thus, even when the interviewee can distinguish between QOL of two different mild conditions, this distinction is not captured by the elicited UCs. TTO results may also be affected by interviewees' perception about life expectancy. In fact, TTO elicitation starts presenting the interviewee with an estimate of his survival, as from national statistics, and a high difference between this value and the interviewee's subjective expectation may bias the UC result.[22]

Thus, we integrated into our system a modification known as daily TTO (DTTO) that should fit better for the above-mentioned cases.[19] Instead of considering the whole lifetime, DTTO assesses how much daily time an interviewee would hypothetically trade in exchange for complete healing. According toBuckingham et al,[19] the UC is computed as:

Zoom Image

where the available daily time is computed as the difference between 24 hours and the interviewee's average sleep duration. While solving the zero-traders problem, DTTO may be unsuitable for unemployed, since they may be excessively willing to give up free time.

Finally, we integrated the willingness to pay (WTP) method,[23] which assesses the percentage of economic resources an interviewee would hypothetically trade in exchange for complete healing. Like DTTO, WTP does not require to face the potentially shocking concept of risk of death or reduced survival, thus being suitable for risk-averse interviewees.


#

Decision Support for Selecting the Elicitation Method

To dispense the interviewer from being familiar with utility theory, we exploited the above-described evidence to develop a DSS for the selection of the elicitation method. The DSS knowledge base is composed of 12 rules ([Table 1]) describing the applicability of the different methods to specific circumstances. Each rule is made up of four components:

Table 1

Rules implemented in the decision support system

Condition

Action

Target

Evidence level

1. Temporary health state

Nonchronic health state, lasting less than 1 year

A

TTO

1

S

DTTO

2

A

SG

3

2. Interviewee is unemployed

Interviewee who is not employed

A

DTTO

4

3. Mild health state

Health state that is not impairing, whose UC is expected to be close to 1

A

TTO

1

S

DTTO

3

4. Short life expectancy

The interviewee's life expectancy is lower than 5 years

A

TTO

2

5. Interviewee is a “caregiver”

The interviewee feels responsible for young children or nonautonomous adult family members

A

TTO

3

A

SG

4

6. Health state involving risky decisions

Impairing health state potentially curable with highly risky operations (e.g., lung transplant, heart transplant)

S

SG

4

7. Patient is zero-trader

Patient refuses to trade/gamble independently of his perception of the examined health state

E

Any method

1

8. Patient's life expectancy expectation differs from the presented value

Patient whose life length expectations differ from the value he is presented with during a TTO task.

A

TTO

3

Abbreviations: DTTO, daily time trade-off; SG, standard gamble; TTO, time trade-off; UC, utility coefficient.


  • Condition: the interviewee's or the health state characteristics triggering the rule (e.g., “unemployed interviewee,” “temporary health state”);

  • Target: the elicitation method the rule refers to;

  • Action: the type of recommendation, i.e., “suggest” (S), “advise against” (A), and “exclude” (E);

  • Evidence level: an integer ranging from 1 (maximum) to 5 (minimum), representing the supporting evidence given by the literature for the suggested action.

For example, the first row in [Table 1] affirms that there is highly reliable evidence against using the TTO for assessing the QOL of a temporary health condition.

Rules are interpreted by an inference engine whose workflow, implemented in Java, is shown in [Fig. 1]. Following the interviewer-initiated DSS activation (A), the system calculates a recommendation for each elicitation method (B). To compute recommendations, the engine splits the triggered rules into rules suggesting the method, and rules advising against it. For each subgroup the engine computes the best evidence level. In case the two subgroups have the same evidence level, the number of rules with that level is used to determine the subgroup to suggest. Recommendations are delivered graphically, exploiting a traffic-light metaphor: green light if the method is suggested and red or yellow light (depending on the reliability of the recommendation) if the use of the method is not recommended. In [Supplementary Fig. S1] (online only) in the Supplementary Material we provide an example of recommendation delivery, complemented with the complete traffic-light color legend ([Supplementary Fig. S2] [online only] in the same file). Leveraging on the received recommendations, the interviewer selects the elicitation method (C). The system might deliver further advices during the elicitation process. For example, during a TTO elicitation, if the interviewee is presented with an estimate of life expectancy that differs from his expectation (D), the interviewer receives a warning and can decide whether to proceed with TTO (E). Testing the zero-trading condition is another key step in the flow (F) since in this case the interviewer is recommended to proceed with another method. At the end of the elicitation, both the interviewee and the interviewer can rate the elicitation experience (G). The score thus collected will provide information on the interviewee's attitude toward the method during future elicitations involving the same subject.

Zoom Image
Fig. 1 DSS workflow: Workflow for guiding the choice of the elicitation method. DSS, decision support system; U, utility coefficient.

#

Integration with Decision Trees

When DTs are used in SDM, they include all the health states the patient might experience as a result of the available treatment options,[16] and every state must be assigned a UC. In this work, we integrated UceWeb with TreeAge Pro,[24] a widely used software for formalizing and running DTs. The TreeAge Pro Object Interface enables to open, update, and analyze DTs using programming languages (Java in our case), thus integrating such functionalities in custom applications. Currently, the UceWeb repository contain two DTs, developed within the EU project Mobiguide,[25] comparing different atrial fibrillation therapies, but it can be easily extended by adding any model formalized with TreeAge Pro.


#

Evaluation

To evaluate the DSS and the overall usability of the system, we asked 51 volunteers to perform elicitations using all the available methods. We selected three health conditions that trigger the rules listed in [Table 1]. In particular, we included a mild disease (mild psoriasis), a painful but temporary impairing condition (rib fracture), and a life-threatening disease potentially curable with high-risk surgery (dilated cardiomyopathy). A fourth health state was “living with an amputated leg,” which does not trigger any rule. Volunteers were enrolled from high school students attending internships at our Engineering Faculty, master students in Biomedical Engineering, and researchers working in our department. Interviewees were asked to answer some socio-demographic questions (e.g., sex, age, caregiving status, and occupation), the answers of which are summarized in [Supplementary Table S1] (online only) in the Supplementary Material. Caregivers were asked to examine leg amputation, since we ideally aimed to isolate the effects of the caregiving status on the obtained UC values. The other subjects were randomly assigned one of the four selected health states. Health states were illustrated to interviewees using text and infographics specifically designed for this study. As an example, in [Supplementary Fig. S3] (online only) of the Supplementary Material we provide an English version of the description used to introduce dilated cardiomyopathy. Then UCs were elicited with all the available methods. After the elicitation, the interviewees filled in the System Usability Scale[26] (SUS) questionnaire, to rate the overall usability of UceWeb. The SUS score computed from the questionnaire ranges from 0 to 100. Moreover, they rated each elicitation method on a scale from 1 (I didn't understand the task/I was very uncomfortable doing it) to 5 (I clearly understood the task/I was very comfortable doing it).


#
#

Results

The overall result of our work is an integrated framework for SDM. Interviewers interested in exploiting DTs for SDM can select a model in the repository. As a response, UceWeb provides the list of the health states considered by the DT. For each state, the interviewer can elicit a UC from the current interviewee. In this case, UceWeb activates the elicitation process, and the interviewer may decide to use the DSS for selecting the best elicitation method. The obtained, personalized, UC values are then set into the DT. In case it is not possible to elicit one or more UCs, the tool anyway provides its user with UC default values, representing population preferences taken from the literature and used to quantify the DT during the model construction step. When all the states are assigned a UC, UceWeb invokes TreeAge Pro methods to perform the analysis. A video illustrating this functionality is available online (https://www.youtube.com/watch?v=_f3NLm8ayCE&feature=youtu.be). The following sections describe the results of the pilot study assessing the interviewees' perception on the tool.

Elicitation Results

We excluded from the analysis those interviewees who did not provide socio-demographic information. The final sample size was 48. For each respondent we collected a socio-demographic profile, five UCs (one per method), and five rating scores (one per method).

First, we analyzed interviewees' willingness to engage in the hypothetical trade. Methods involving the death concept (TTO and SG) showed a higher number of zero-traders (11 and 6, respectively) compared with DTTO (0) and WTP (3). According to the literature,[17] this effect was more frequent when evaluating temporary (like rib fracture) and mild (like mild psoriasis) health states ([Supplementary Table S2], online only). For such states, interviewees' reluctance to trade in TTO and SG is reflected also by the resulting UCs ([Supplementary Table S3], online only): coefficients elicited through SG and TTO were significantly higher than those obtained with other methods (Friedman test p-value < 0.005). Paired Wilcoxon signed-rank tests were performed to compare each pair of elicitation methods. Test results regarding rib fracture suggest that the median TTO value (0.98) was significantly higher than those of DTTO (0.78) and WTP (0.94) (p-values of 0.034 and 0.043, respectively), and that SG (median value: 0.99) produced significantly higher values than DTTO (p-value: 0.027). Test results for mild psoriasis indicate that all the median UCs from a method involving the death concept (UCTTO = 0.79, UCSG = 0.8) are significantly higher than those from any other method (UCDTTO = 0.68, UCWTP = 0.61) (p-value < 0.05 for any pair comparison). Due to interviewees' reluctance to trade when evaluating rib fracture and mild psoriasis, the distributions of the UCs elicited through TTO and SG are skewed toward 1, with the ceiling effect being particularly evident for rib fracture ([Supplementary Fig. S4], online only).


#

Usability Assessment

SUS scores ranged from 47.5 to 100 (interquartile range: 67.5-88.75; median: 78.75). Since SUS scores above 68 are conventionally considered above average, we conclude that interviewees considered UceWeb as a user-friendly application. Moreover, they assigned positive rating scores to all the elicitation methods. Median rating scores ranged from 4 (RS, SG, TTO, and DTTO) to 5 (WTP).

Despite the overall good usability, we highlighted some criticalities. The DTTO task was difficult to understand. During our experiment, interviewees needed help to understand the terms of the DTTO trading process, and struggled to interpret the proposed extra sleeping time as an unfavorable tradeoff.[19] However, with the due explanations, interviewees seemed more confident with DTTO than with TTO. Consequently, no interviewee was zero-trader in DTTO, DTTO UCs were lower than TTO UCs, and no ceiling effect was detected.


#
#

Discussion and Conclusion

The new version of UceWeb is an integrated environment for collecting and exploiting QOL data in SDM. To our knowledge, UceWeb is the first tool that integrates functionalities for eliciting UCs with both a UC repository and the facilities for running custom, personalized DT models. Besides being useful for SDM, UceWeb may be used to collect UCs within an international network. Since the elicited UC values are complemented with the patient's (anonymous) profile, researchers interested in CUA can retrieve from our repository specific sets of UCs that are appropriate for their target population.

Our set of elicitation methods is the widest among the tools known in the literature and showed satisfactory usability during our study. UceWeb is also the first tool including a DSS for selecting the elicitation method. The UCs collected in our experiment confirm the rules implemented in the DSS. In particular, the temporary or mild nature of the health condition led our interviewees to be reluctant to trade when using TTO or SG, independently of the perceived desirability of the condition.

We are aware of the limitations of our study. First, we did not implement all the variants of the SG and TTO methods. For example, a multistep variant of the TTO known as “chained TTO” may be suitable to value temporary states, but in the literature we found a lack of agreement on how to perform such elicitation.[27] We are also aware of the two variants of the TTO described by Artaso and Díez.[28] In the first one (“TTOqol”), interviewees trade QOL in exchange for life expectancy. In the second one (“TTOequiv”), interviewees have to compare two health conditions (A and B) by setting “time equivalence,” i.e., the amount of time spent in A that is equivalent to a given amount of time in B, according to their perception. These methods may be added in the future, after collecting enough information from the literature to add rules on their applicability into the DSS knowledge base. In addition, it will be also possible to explore the use of indirect methods that ask the patient to fill in a health-related QOL questionnaire (e.g., Euroqol, https://euroqol.org/) and compute a UC value as a weighted sum of the given answers.[29]

Another limitation is that we only rely on DTs as formalism for clinical decision-making. However, also influence diagrams could be used, and some literature suggests that they may be more flexible in managing multiple time-distributed decisions.[30] [31] Thus, we will evaluate the possibility to integrate UceWeb with tools for running such models. An additional limitation is that TreeAge Pro is a commercial product that requires a license. Currently, UceWeb is in a research-development phase. To provide the same functionalities as a service, an agreement with the TreeAge Pro company must be achieved.

Finally, as regard the interviewed sample, the low number of caregivers did not allow us to draw definite conclusions about the impact of caregiving on the elicitation process.

UceWeb is currently focused on the aspect of SDM that involves evaluating patient's preferences related to a set of health conditions, with the goal of selecting the decision option that maximizes a specific payoff, the QALYs. This is indeed just one aspect of application of SDM, which has also been the focus of multiple-criteria decision analysis techniques. These methods rely on the evaluation of patients' preferences to rank the multiple criteria that have to be taken into account to compare different decision options.[32] [33] In the future, we could consider this as an interesting extension of UceWeb to provide the possibility of a personalized selection of the payoff.

To test UceWeb further, we are planning additional experiments, in which we will enroll more caregivers. In addition, we will plan a pilot study to collect feedback on our tool also from interviewers, since this first test focused only on the interviewee's perspective. We will also enroll clinical experts to assess how they perceive the functionalities that are specifically dedicated to the medical decision process (e.g., the one for running DTs).


#
#

Conflict of Interest

None declared.

Supplementary Material


Address for correspondence

Elisa Salvi, MS
Department of Electrical, Computer and Biomedical Engineering, University of Pavia
Via Ferrata 5, 27100, Pavia
Italy   


Zoom Image
Zoom Image
Zoom Image
Fig. 1 DSS workflow: Workflow for guiding the choice of the elicitation method. DSS, decision support system; U, utility coefficient.