The oracle of D-A-Ch – Results of a Delphi Survey for the development of the evidence- and consensus-based tool “My Logbook”

Abstract

Objective The present report describes the results of four delphi surveys conducted within the quality improvement project “My Logbook” which aims to translate evidence-based standards for psychosocial care in pediatric oncology into a practical consensus-based tool.

Methods In four consecutive delphi surveys a total of n=153 international, multi-disciplinary experts rated the content, method, and design of the different booklets of “My Logbook” which a local expert group had conceptualized. After each survey, the feedback was incorporated, and the changes were evaluated in a final consensus vote by the quality assurance panel of the PSAPOH.

Results While some surveys led to a review on a page level, most booklets as a whole reached the consensus-level of approval. Over the course of the surveys, any revisions and comments were incorporated in the booklets, and approval rates increased steadily.

Discussion The delphi surveys ensured the integration of multi-disciplinary, international expertise, uncovering issues such as language barriers and the need for a user manual that would not have been evident in first line . The incorporation of the input led to a continuous improvement of the tool, reflected in steadily increasing acceptance rates in the consecutive survey rounds.

Conclusion The incorporation of the expert input as well as the additional development of a user manual resulted in a final version of the “My Logbook” apt for the interdisciplinary application in pediatric oncology in the entire DACH-region.

Zusammenfassung

Hintergrund Es werden die Ergebnisse von vier Delphi-Befragungen vorgestellt, die im Rahmen des Qualitätsverbesserungsprojekts "Mein Logbuch" durchgeführt wurden. Das Projekt zielt darauf ab, evidenzbasierte Standards für die psychosoziale Versorgung in der pädiatrischen Onkologie in ein praxisorientiertes, konsensbasiertes Instrument zu übersetzen.

Methode In vier aufeinanderfolgenden Delphi-Befragungen bewerteten insgesamt n=153 internationale, multidisziplinäre Expert:innen den Inhalt, die Methode und das Design der verschiedenen "Mein Logbuch" Themenhefte, welche zuvor durch eine lokale Expert innengruppe konzipiert wurden. Nach jeder Befragung wurden die Rückmeldungen eingearbeitet und die Änderungen in einer abschließenden Konsensabstimmung durch die Fachgruppe Qualitätssicherung der PSAPOH bewertet.

Ergebnisse Während bestimmte Ergebnisse von Delphirunden zu einer Revision auf ausgewählten Themenseiten führten, erreichten die meisten Themenhefte insgesamt einen akzeptablen Konsensus. Sämtliche Rückmeldungen und Anmerkungen wurden in die Themenhefte eingearbeitet, wodurch Zustimmungsraten stetig stiegen.

Diskussion Die Delphi-Befragungen gewährleisteten die Integration von multidisziplinärem, internationalem Fachwissen und deckten Themen wie Sprachbarrieren und die Notwendigkeit eines Benutzerhandbuchs auf, welches ursprünglich nicht vordergründig priorisiert wurde. Die Einbeziehung des Inputs führte zu einer kontinuierlichen Verbesserung des Instruments, was sich in stetig steigenden Zustimmungsraten in den aufeinanderfolgenden Befragungsrunden widerspiegelte.

Schlussfolgerung Die Einarbeitung des Expert:inneninputs sowie die zusätzliche Entwicklung eines Benutzerhandbuchs führten zu einer finalen Version des "Mein Logbuch", geeignet für den interdisziplinären Einsatz in der pädiatrischen Onkologie in der gesamten DACH-Region.

Keywords

delphi survey - quality improvement - integrated care - pediatric oncology - consensus-based research

Schlüsselwörter

Delphi-Befragung - Qualitätsverbesserung - integrierte Versorgung - pädiatrische Onkologie - konsensus-basierte Forschungpublished online 2023

Introduction

Since psychosocial care has been identified as a fundamental part of pediatric oncology, an increasing number of guidelines and standards for quality and homogeneity has been established [1] [2] [3]. However, to date there are only few consensus-based, standardized processes to ensure the implementation of evidence-based care and standards. Hence, there is still a low number of medical centers verifiably providing the required psychosocial care, with a great variability in the extent to which standards are fulfilled [3] [4] [5]. Major obstacles impeding the implementation of guidelines and the use of the manifold evidence-based psychosocial methods include diverging priority setting in the multi-disciplinary context, as well as the shortage in financial and personnel resources, with sparsely developed psychosocial services, lacking systematic documentation and regular supervision [6] [7]. Furthermore, for interventions to be feasible, and apt for the complex, multi-professional field of pediatric oncology, they also need to be developed in this specific context and in collaboration with all relevant stakeholders [6] [8]. As stated in guidelines such as the Consolidated Framework for Implementation Research (CFIR), it is essential to use methods such as delphi surveys to involve especially the target population when aiming to provide consensus-based patient-oriented care [3].

To bridge this persisting gap between research and practice, the Quality Improvement Project "My Logbook! – I Know my Way Around!" (Trial registration identifier: ClinicalTrials.gov Identifier: NCT04474678.) uses a multi-stage process of Plan-Do-Study-Act (PDSA) cycles [9] to translate the evidence-based S3-guideline for Psychosocial Care in Pediatric Oncology and Hematology [2] into a practical psycho-educative program for children in oncological care [10]. The resulting tool consists of various booklets, each focusing on one topic and the corresponding psychosocial stressors encountered by patients during cancer treatment (e. g., “ABC of chemotherapy”, “Mission stem cell transplantation”). It thereby aims to support professionals to provide adequate care and ensure that patients are accompanied by high-quality psychoeducation, preparation, training, and interventions throughout the entire treatment process.

This brief report describes the delphi survey as one element of the quality improvement project, which enabled the tool to be evaluated by a multi-professional expert group before applying and evaluating it in a multi-center pilot study. The aim is to use the expertise of health care professionals, to ensure that the material of the project portrays the current position of psycho-social care in pediatric oncology and hematology and therefore represents a common standard that can be used across German-speaking countries.

Methods

Delphi surveys were chosen as the appropriate method for the evaluation and continuous consensus-based construction of the novel tool based on the professionals’ specific expertise. One key benefit is that feedback can be directly incorporated into the developed tool and constant improvement can be achieved via repeated evaluation and integration phases [11].

Panel member selection

Potential survey participants were contacted through the mailing list (250 health-care professionals) of the Psychosocial Association in Paediatric Oncology and Haematology (PSAPOH) a working group of the Society for Pediatric Oncology and Hematology (GPOH), which is an expert organization operating in German-speaking countries. The professionals were also encouraged to disseminate the e-mail within their work group. The prerequisites for joining the PSAPOH are specific qualification, ongoing professional activity as well as the recommendation through an existing member. Therefore, participants’ specific expertise and qualification was assumed based on the membership in this expert organization. Primarily, psychosocial experts were invited to participate in the survey in order to provide professional input on the psychosocial content. At the same time, experts with diverse levels of experience were included to create a heterogeneous group of respondents. Additionally, experts from other disciplines such as medicine and nursing were invited, to improve and promote interfaces and interdisciplinarity in pediatric oncology. Based on a descriptive questionnaire, the underlying expertise can be assigned and care was taken to ensure that the weighting of the expertise has a psychosocial focus. It must also be taken into account that the Delphi survey represents a second step in the development process and that a specific request for expertise was addressed in the composition of the panels for the conceptualization of the thematic booklets. The specific profession as well as the duration of activity was assessed at the beginning of the online questionnaire. A minimum of n=40 participants was striven for in each survey round.

Procedure and survey modality

The booklets approved by the local experts were evaluated in one or more of four consecutive delphi rounds which took place online from November 2017 to March 2018, September 2018 to January 2019, November 2020 to February 2021, and from May to July 2022. All questionnaires were constructed over the platform Sosci survey and followed the same structure to allow comparisons between the survey rounds. The surveys were conducted asynchronously to allow pseudo-anonymity and participants had the option to interrupt and save their response. All questionnaires Each booklet section and any supplementary material was evaluated with the following factors: “I agree”, “Minor revision” (formal errors, harmonization of language differences between Austrian, German and Swiss German), “Supplemental Information” (additional material, explanations, information which are useful, but not essential for a minimum standard), and “Major revision” (major issues related to content). An example page for the evaluation of booklets and the accompanied manual is shown in [Fig. 1]. Due to the close collaboration with experts on the subject matter during the development of the booklets, the overall relevance of the topic was not called into question.

Fig. 1 Example page for the evaluation of a booklet-section and accompanied manual.

The collected feedback was analyzed quantitively and qualitatively. In accordance with the S3-guideline [2], the satisfactory level of consensus was determined by 80% including both categories “I agree” and “Minor revision”. For a final decision, booklets failing to reach the 80% mark were presented to the PSAPOH quality assurance panel (Fachgruppe Qualitätssicherung der PSAPOH). After each delphi round, the feedback was anonymized and prepared to be presented in the consensus-meetings as well as to all participants. The feedback was then incorporated by the local expert group to continuously develop the booklets. These changes were discussed and evaluated in a final consensus vote by the PSAPOH quality assurance panel. Updates, including current results and the acknowledgment of participating centers and experts were disseminated to the community via the newsletter. The “My Logbuch” project containing the presently described delphi surveys, was approved by local Ethics Committees of the Meidcal University of Vienna and written consent was given by all participating experts.

The process of the entire project is described in more detail and visualized in [Fig. 1] of the corresponding protocol paper [10].

Results

Panel members

In total there were n=81 participants (n=153 responses), n=25 answered the first round of the delphi-survey, n=41 the second round, n=45 the third round and n=42 the fourth and final round. The respondents´ professions encompassed (clinical) psychologists (55%), psychotherapists (14%), medical doctors (8%), nurses (8%), social pedagogues (8%) educators (6%), art/music therapists (6%), and survivors (6%). The duration of occupation in the area of pediatric oncology was distributed quite equally ranging from 0 to over ten years. The majority of participants stated to primarily work in acute care (at least 71% in each survey). Anonymity was offered to all participants but only utilized by the manority. See Table 1 in the supplemental material for a detailed description of the participants in each survey round.

Delphi surveys

[Fig. 2] shows an overview of all evaluations at the booklet-level. The booklets were widely accepted and only the first booklets needed more than one delphi round to reach a final version. Most comments were related to formal errors and language use. Four booklets that were evaluated in round 1 were revised and revaluated in round 2 along with additional, newly developed booklets. Even though the evaluation for the booklets as a whole was sufficient, the evaluation at the section level did not meet the satisfactory level of 80% consensus consistently. The only booklet that did not achieve the 80%-level of consensus in the overall evaluation was “Mission stem cell transplantation” in round 4. For detailed results for each delphi survey see [Fig. 2], as well as Table 2 in the supplementary material. The evaluation was positive in general, still, there was important feedback to incorporate in the booklets. [Fig. 2] presents an overview on all booklets that were evaluated. A more detailed overview can be found on the GPOH website (https://www.gpoh.de/sites/gpoh/kinderkrebsinfo/content/e1676/e176475/e176588/e260155/PsychosozialeBasisversorgung_MeinLogbuch_Oncobasicvers.2.0.1_2022–08–30_ger.pdf).

Fig. 2 Overview of evaluations at the booklet-level. The horizontal line marks the level of consensus of 80%. *Booklets were part in the survey repeatedly.

1^st round

During the first delphi round parts of four out of five booklets did not meet the acceptance level of 80%. The booklet “Everything there is to know!” building a therapeutic relationship was evaluated to need a major revision from over 20% of participants for each section. There was a lot of feedback concerning language disparities, but also relating to the content (e. g., “only ideal-typical families are depicted in the material”). Some of the interactive methods in the booklet were not in terms of explantion of usage for participants and they commented that not all professions working in pediatric wards were depicted (major revision: section 1=36%, section 2=32%, section 3=44%, section 4=48%, section 5=24%, section 7=36%).

Feedback for the booklet “My illness” was related to better describing concepts and the level of detail (major review: section 2=36%, section 4=36%, section 7=40%). The booklet “I’M Ready” (preparation for MRI) was evaluated as focusing too much on written text and not being and interactive enough, because it mostly covered issues related to (major review: section 2=36%). “My path through radiotherapy” was generally evaluated well, but some parts of the material were not clear to participants (major review: section 6=36%).

2^nd round

The second round of the delphi survey consisted of ten different booklets, which also included revisions of four booklets from the first round. In general, the revisions made with regard to methodology (e.g. more precise descriptions in a manual, reduction of complexity) were positively received, resulting in a significant increase in the acceptance level compared to the first round. Also in this round we detected details within the generelly high rated booklets to adress for review. For the booklet “Everything there is to know!” the list of health-care professionals working at pediatric wards was updated to include educators and remedial teachers (major review: section 5=23%). For the newly designed booklet “My path to diagnosis” a more child-appropriate way of explaining certain words was recommended and psychoeducational section of the booklet was generally evaluated as being too complex (major review: section 1=27%). Feedback concerning “ABC of chemotherapy" related to unclear language of biological processes and not mentioning certain side-effects (major review: section 2=24%). Both “I’M Ready” (major review: section 2=37%) and “My plan for rehab” (major review: section 1=32%) were evaluated as being too clinic specific (e. g., “it is not possible for a parent to be present at a child’s MR-scan in every clinic”). Similar feedback was also given to “Managed! – My follow-up care”, which some participants considered as too deficit-oriented (major review: section 2=22%, section 4=39%, section 5=34%).

3^rd round

Generally, the booklets in this round were evaluated very favourably, only in one booklet (“My check-ups”, major review: section 2=36%), the high complexity was adressed as an issue.

4^th round

Again in this round all booklets reached an acceptable level of consensus. The detailed evaluation showed apart from the “Parent’s information” booklet, considering feedback on design and layout (major review: section 2=21%), “Mission stem cell transplantation” was noted for being unclear on biological processes and using different terms than other clinics are used to. This booklet as a whole did not meet the level of consensus (major review: section 2=29%, total=21%).

Discussion

The delphi surveys represent an invaluable part of the development of the “My Logbook”, since they ensured the integration of expertise of multi-disciplinary international professionals in the construction of the tool rather than only in the evaluation. The comparatively high number of revisions at the beginning of the process depicts the amount of expertise and input that could only be provided by the extended expert team, but not by the local experts to adequatly adress regional an clinicla specific needs. This highlights the importance and benefits of implementation research, integrating the various perspectives of all relevant stakeholders from the start of the project on [3]. In contrast, the steady decrease in requested major revisions indicates a continuous improvement in basic concept and structure of the drafts based on the experience gathered in the preceding delphi rounds. The overall positive feedback indicates a general openness of international professionals to implement the tool in clinical practice, which will be further evaluated in the multi-center pilot study.

One major issue that only became evident during the international surveys were releated to regional german speaking disparities differences in terminologies. While the first versions of the “My Logbook” booklets were developed only for the use in the local Austrian pediatric neurooncological department, later versions were adapted to be applicable in the entire field of pediatric oncology in all German-speaking countries. This required for considerable adaptations in the language used in the booklets to be apt for all dialects spoken in the DACH-region as well as a generalization of technical terms used. Other changes regarded the structure and design of the booklets leading to a restriction of the informational part to a maximum of two pages; clearly visible changes in the design, the structure of the individual pages, and the graphical content; as well as the introduction of additional booklets for topics insufficiently covered by the proposed booklets.

Expanding the project from Austrian pediatric neuro-oncology to the adaptation for pediatric-oncology in the entire DACH-region also lead to clear changes in the and reduction of complexity of the booklets. While the first versions were rather narrow and mostly contained informational material and fill-in exercises, the new versions provide a multitude of different methods and interventions such as instructions for handicraft exercises, puzzles, and riddles. To account for the heterogeneity of the field of pediatric oncology in German-speaking countries, more recent versions use more consistent language and have a higher emphasis on interdisciplinary linkage. Although this change implies a considerably improvement in the quality of the tool, it also made it less self-explanatory and the need for a manual became evident. Hence, a “My Logbook” manual for health care professionals was developed which contains basic information on the concepts of the S3-guideline underlying the “My Logbook”, as well as specific instructions for its application, including sample answers and suggested material [12]. In the realm of the ongoing multi-center study, “My Logbook” material can be provided to interested professionals and institutions upon reasonable request to the corresponding author and are described in further detail in the corresponding protocol paper [10].

Limitations

The “My Logbook” project grew from a local initiative to an interdisciplinary multi-centered study. Although the approach is contrary to traditional scientific research with clearly predefined research questions and aims, in this project the rationale of CFIR and PDSA cycles were followed to build a consensus-based and patient-oriented tool. While in the present project, the extended expert group already represented most disciplines and geographic areas, a continuous effort is necessary to ensure that this process-oriented tool is continuously adapted to the various perspectives in the constantly changing system network of pediatric oncology in the DACH-region. We did not have a constant panel of survey participants which should be considered by future studies. Furthermore, in the present study, the participants’ expertise was assumed based on their PSAPOH membership while in future studies, the level of expertise should be assessed in terms of experience, professional domain, specific (research) focus, additional qualifications, etc. to ensure that feedback on the specific matter is exclusively given by qualified topic experts. In the current study, we decided to keep the survey open to enable experts of specific topics within pediatric oncology to participate in a single delphi round (e. g., experts in stem cell transplant or family-oriented rehabilitation). However, future studies should enforce a better balance between different disciplines and patient experts.

Conclusion

The delphi survey as part of the “My Logbook” project represents an essential step in integrating ample expert perspectives into the development of the tool. The manifold input led to a steady improvement and adaptation of the tool, resulting in a version apt for the application in the multi-disciplinary field of pediatric neuro-oncology in the entire DACH-region. Especially changes in language, structure, and content as well as the additional development of a user manual contribute to the quality of interventions provided as well as better qualification and objectivity in the health care providers.

Contributor’s Statement

L. Weiler-Wichtl conceptualized, designed, and supervised the project, created the graphics, drafted, and revised the manuscript, and acquired funding. A. Kollmann, V. Fohn-Erhold, C. Schneider and U. Leiss substantially supported the conceptualization, design, and organization of the project. V. Fohn-Erhold, V. Rosenmayr, M. Hopfgartner, and J. Friess were involved in the acquisition of the data. Fohn-Erhold, M. Hopfgartner, and J. Friess were involved in the analysis and interpretation of the project the protocol describes. R. Hansl and K. Herzog contributed to editing the manuscript. All authors read and approved the final manuscript and agreed to be accountable for their contributions.

Conflict of Interest

The authors declare that they have no conflict of interest.

Acknowledgements

We want to thank all the experts who participated in the delphi survey for their constructive feedback and dedicated collaboration. All names are listed in (Weiler-Wichtl & Fohn-Erhold, 2021). Reference: Weiler-Wichtl, L. J., & Fohn-Erhold, V. (2021). “Mein Logbuch – ich kenn mich aus!” – Das Handbuch (COPE Group, Ed.; ONCO Basic).

• References

• 1 Jacobsen PB. Clinical practice guidelines for the psychosocial care of cancer survivors: Current status and future prospects. Cancer 2009; 115: 4419-4429
• 2 Schröder HM. et al. Psychosoziale Versorgung in der Pädiatrischen Onkologie und Hämatologie - AWMF S3 Leitlinie Register Nr. 025/002. Gesellschaft für Pädiatrische Onkologie und Hämatologie (GPOH). 2019
• 3 Tyc V, Fairclough D, Fletcher B. et al. Children’s distress during magnetic resonance imaging procedures. Child Health Care 1995; 24: 5-19
• 4 Scialla MA, Canter KS, Chen FF. et al Delivery of care consistent with the psychosocial standards in pediatric cancer: Current practices in the United States. Pediatr Blood Cancer 2018; 65 10.1002/pbc.26869
• 5 Weis J, Giesler JM. Standards in der psychosozialen Versorgung von Krebspatienten. Onkologe 2016; 22: 198-205
• 6 Baxter S, Johnson M, Chambers D. et al. The effects of integrated care: a systematic review of UK and international evidence. BMC Health Serv Res 2018; 18: 350
• 7 Giesler J, Faust T, Ernst J. et al. Ausgewählte Merkmale der Struktur- und Prozessqualität der psychosozialen Versorgung in ambulanten psychosozialen Krebsberatungsstellen in Deutschland. Eine Bestandsaufnahme. Gesundheitswesen 2019; 81: 336-343
• 8 Munson SA, Friedmann EX, Osterhage K. et al. Usability Issues in Evidence-Based Psychosocial Interventions and Implementation Strategies: Cross-project Analysis. J Med Internet Res 2022; 24: e37585
• 9 Schurman JV, Gayes LA, Slosky L. et al. Publishing quality improvement work in Clinical Practice in Pediatric Psychology: The ‘why’ and ‘how to’. Clinical Practice in Pediatric Psychology 2015; 3: 80-91
• 10 Weiler-Wichtl LJ, Fohn-Erhold V, Schneider C. et al. Bridging the gap: A quality improvement project to implement psychosocial care standards into clinical practice in pediatric oncology. Klin Padiatr 2023; 235: 350-359
• 11 Niederberger M, Spranger J. Delphi Technique in Health Sciences: A Map. Front Public Health 2020; 8: 457
• 12 Weiler-Wichtl LJ, Fohn-Erhold V. Mein Logbuch - ich kenn mich aus!- Das Handbuch, ONCO Basic. C. Group. 2021

Publication History

Article published online:
04 December 2023