Uptake of an Integrated Electronic Questionnaire System in Community Pediatric Clinics

Abstract

Objective The study aimed to evaluate an integrated electronic questionnaire system implementation in outpatient community pediatric practices on workflow, completion rates, and recorded scores.

Methods We evaluated the implementation and outcomes of an integrated electronic questionnaire system at 45 community pediatric practices that used standardized questionnaires to screen for autism, depression, and substance use and to measure asthma control. Electronic health record (EHR) data for all well child visits were extracted for the 3 months before and after implementation. We used statistical process control charts to evaluate questionnaire completion rates and Chi-square tests to evaluate screening completion and positive screening rates. The collection and entry of questionnaire information was observed and timed.

Results EHR data included 107,120 encounters across 45 practices that showed significant and sustained improvement in completion rates for all questionnaires. The rate of recorded concerning questionnaires decreased for asthma control (19.3 vs. 12.8%, p < 0.001), stayed the same for autism (96.6 vs. 96.2%, p = 0.38), decreased for depression (9.5 vs. 6.7%, p ≤ 0.001), and increased for any substance use (9.8 vs. 12.8%, p < 0.001). Twelve practices were observed, and patient time and staff time managing questionnaires were decreased after implementation.

Discussion Electronic questionnaire administration saved staff time and patient time. We report overall improvement in questionnaire completion rates, with notable variation in improvement in completion across practices and in change in concerning recorded result rates across measures.

Conclusion Conversion of four standard paper questionnaires to an integrated electronic system reduces patient and staff time while increasing completion rates when well integrated into routine care.

Keywords

asthma - autism spectrum disorder - depressive disorder - surveys and questionnaires - electronic health records

Background and Significance

The current recommendations from Bright Futures and the America Academy of Pediatrics (AAP) for preventive pediatric health care include 30 different activities at different intervals, including specific biometric measurements, sensory screening, immunizations, and developmental/behavioral screening.[1] Several of the recommended developmental and behavioral health screens require parent or child questionnaires. These include maternal depression screening (through 6 months of age), autism spectrum disorder screening (at 18 and 24 months), and adolescent depression screening (annually from age 12).[1] Additionally, screening for depression and documenting a follow-up plan is a Healthcare Effectiveness Data and Information Set (HEDIS) measure for adolescents age 12 to 17 years old.[2] Specific payers or health systems may also incentivize completion of other specific questionnaire screening. In addition to recommendations for all routine preventive visits, additional disease-specific questionnaires are important components of chronic disease management, such as the Asthma Control Test used for asthma.[3] However, screening rates for behavioral health and asthma control at well child care (WCC) visits remain suboptimal.[4] [5] [6]

Electronic questionnaire capture is an appealing strategy for improving questionnaire completion rates.[7] [8] When integrated into EHRs, electronic questionnaires can reduce time and avoid potential errors from staff data entry. Electronic questionnaires can also offer immediate validation and interpretation. Additionally, using electronic questionnaires can allow computer algorithms to parse complicated rules about which child needs which questionnaires. Finally, by avoiding paper, electronic questionnaires can provide greater confidentiality, ensure that results are given to the correct clinicians, and reduce administrative and space requirements for copying and storing paper questionnaires.

However, there are persistent concerns that electronic questionnaire implementation is difficult and disrupts clinical workflow. Costs associated with software and hardware may dampen enthusiasm for the transition. Training staff and clinicians to adopt a new system is difficult and time consuming. Storing, charging, and sanitizing tablets require new space arrangements and workflow. Given these concerns, it is essential to know the potential short-term and long-term benefits and harms: can practices expect to improve questionnaire completion rates? Will rates of positive and negative screens change? How will workflow before and after compare? Currently, the data available to answer these questions are limited, with many studies either from other countries or within subspecialty pediatric clinics.[9] [10] [11] [12] [13] To allow stakeholders to make informed decisions about the adoption of integrated electronic questionnaire systems, updated and pediatric-specific data on the impact of transitioning to electronic questionnaires are needed.

Objective

To examine the implementation of an integrated electronic questionnaire system in 45 outpatient community pediatric practices, following the Institute for Healthcare Improvement's Model for Improvement. In April 2018, all practices transitioned to collecting multiple questionnaires through tablets, allowing examination of impact on questionnaire completion rates, positive screening rates, and workflow.

Methods

Setting

Children's Community Pediatrics is 45 pediatric primary care offices that are part of UPMC Children's Hospital, a top 10 ranked academic children's hospital. Practices use a common EHR (Epic), and practice size ranges from 2 to 40 clinicians and 1,200 to 11,000 patients. Geographically, the practices span 16 counties, including 6 urban counties and 10 rural counties. As the sixth most populous state, Pennsylvania child racial and poverty demographics are similar to U.S. child demographics but have a slightly higher proportion of children living in rural areas.

Patient-Reported Questionnaires

A set of standardized questionnaires were considered routine practice at WCC visits across all practices. These include the Modified Checklist for Autism in Toddlers (M-CHAT; 18 and 30 months) 20-item questionnaire to screen for autism,[14] the Patient Health Questionnaire (PHQ)-2 (ages 11 and older) 2-item questionnaire to screen for depression, and the Screening to Brief Intervention (S2BI) (ages 11 and older) 3-item screen about alcohol, tobacco, and marijuana.[15] The PHQ-9 was administered if the PHQ-2 was positive. If the 3-item S2BI was positive, it is continued for 12 other substances. For children with asthma, an age appropriate Asthma Control Test (ACT) was to be administered at WCC visits (child version for ages 4 to 11 years and the adult version for ages 12 years and older) which is a 5-item questionnaire about asthma symptoms.[16] [17] [18]

Pre-Implementation Workflow

Before implementation, most questionnaires were collected on paper; front desk staff would determine if a patient was present for a WCC visit, determine appropriate paperwork including asking patients if they have asthma, and give patients and their families the papers to complete in the waiting room. To become discrete data within the EHR for monitoring and analysis, the paper answers needed to be transcribed into the “flowsheet” section of the EHR, usually performed by a medical assistant during rooming. The exception to this collection mechanism was the M-CHAT. M-CHAT had been collected by tablets for 4 years. An icon appeared on the check-in staff's desktop to indicate that a tablet computer should be given to the patient's parent for completion of the M-CHAT in the waiting room. These results were immediately available within the EHR. Thus, during the pre-implementation period, tablets were only given to patients for two specific WCC visits (18 and 30 months) when the M-CHAT was collected.

Implementation Details

The decision to transition the PHQ, S2BI, and ACT to an electronic process was made by an interdisciplinary, cross-practice quality, and safety team; the EHR patient-reported outcomes team constructs questionnaires for all departments and was paired with the Pediatric service line's EHR physician champion, quality analysts, and operations staff. Each of these groups uses the Institute for Healthcare Improvement's Model for Improvement to continually improve patient safety and practice efficiency.

The assignment rules for different questionnaires replicate the recommended questionnaire procedures before implementation. Most questionnaires were assigned at WCC visits by using rules based on age. The ACT was only assigned if asthma was recorded on the patient's problem list. If questionnaires were assigned to a patient, there was an icon on the check-in staff's desktop to indicate that a tablet should be given to the patient (or accompanying parent or caregiver) for completion in the waiting room. If the questionnaires were not completed before rooming, the rooming staff have an “answer questionnaires” section within the rooming section of the EHR where the questions can be asked verbally to the patient (or parent). Answers are stored immediately and directly into the EHR where they can be tracked over time, pulled into documentation, and used to generate alerts (e.g., an alert to the clinician if a patient screens positive for depression).

Practices were given tablets by their patient volume (total number of patients on their busiest day divided by 15). All practices decided to have tablets completed in their waiting rooms. All practice staff were invited to attend a live online 1-hour training session with EHR operations staff. This training was recorded and available for those who could not attend the live training. On the day of implementation, operations staff was available by phone to help troubleshoot any problems in real time. Front desk staff was trained to store tablets in a locked cabinet overnight with chargers, add the questionnaire indicator to their registration screen, bring up a patient's questionnaires by entering the encounter number onto the tablet, and to clean the tablet after each use. EHR operations also scheduled an open-invite check-in 2 and 6 weeks after implementation to troubleshoot any new issues.

Evaluation

This implementation was designed to reduce paperwork and administrative time, reduce data entry errors, and improve the number questionnaire answers recorded in the EHR. To assess paperwork and administrative time, a research assistant observed a half-day at 12 randomly selected practice waiting rooms before and after implementation. Before implementation, they recorded if the patient needed paperwork, the amount of time spent by staff preparing paperwork, the time spent by patients/parents completing paperwork, and if the paperwork was abstracted into the chart before the patient saw the clinician. Practices were also interviewed about how long it takes them to prepare paper forms for the day and if paper forms are abstracted into the chart. The same practices were observed 6 to 8 weeks after implementation for the number of patients who needed paperwork or a tablet, the time spent by patients/parents completing the paperwork or tablet, and interview of staff about their experience with the tablets.

To evaluate the number of questionnaire responses entered in the EHR, we extracted data from the EHR from the 3 months before go-live (spring 2018), allowed a 6-week adjustment period, and 3 months of data after adjustment (summer 2018) for all 45 practices. The data are from all WCC visits (ages 1 and over) that were completed, arrived, or unresolved and include encounter date and location, age of the child at the encounter, questionnaire assignment indicators (e.g., an indicator which is generated by the EHR), and questionnaire scores. For questionnaires administered by paper and flowsheet, we collected scores from 1 day before the encounter to 7 days after in case data transcription occurred after the encounter. For questionnaires administered by tablet, we collected scores on the day of encounter.

Analysis

To examine the impact of implementation on workflow, we examined workflow data using descriptive statistics. To assess impact of implementation on questionnaire completion over time, we used statistical process control charts focused on 1-week interval. To examine summative impact, we compared pre-implementation and post-implementation EHR data, including visit volume, screen completion rates, and positive screen rates, using Chi-square tests. Finally, to understand variation in impact at the practice level, we examined variation in pre-implementation and post-implementation screening rates across practices.

Results

Electronic Health Record Data

There were 107,120 encounters across 45 practices ([Table 1]). More WCC visits occurred during the post-implementation period (n = 65,670) than the pre-implementation period (n = 41,450) because the post-implementation data were extracted in the summer when more WCC visits occur. The number of WCC visits per site varied from 103 to 1,900 during pre-implementation and from 270 to 2,665 post-implementation.

Table 1 Descriptive information about well child care visits

	Number of pre-implementation visits, (%)	Number of post-implementation visits, (%)
Total visits	41,450	65,670
Age group
12–23 mo	7,996 (19)	8,219 (13)
2–5 y	12,505 (30)	14,214 (22)
6–10 y	9,419 (23)	14,338 (22)
11–14 y	6,263 (15)	14,666 (22)
15–17 y	3,911 (9)	10,111 (15)
18+ y	1,356 (3)	4,122 (6)

We observed sustained improvement in rates of completion for each of the four questionnaires coinciding with implementation ([Fig. 1]); additional details of completion rates and positive screen rates are discussed for each questionnaire below.

Fig. 1 Statistical process control charts focused on 1-week intervals for completion rates for the four questionnaires. Panel A is the Modified Checklist for Autism in Toddlers, panel B is the Patient Health Questionnaire-2, panel C is the Screening to Brief Intervention, and panel D is the Asthma Control Test.

Modified Checklist for Autism in Toddlers

Of the four questionnaires, the M-CHAT is the one that had electronic collection before the broader implementation, but completion rates were suboptimal at 75% for the 18 months and 68% of 30 months WCC visits. With broader implementation of tablet use for multiple questionnaires and age groups, we observed an improvement in completion rates of the M-CHAT ([Table 2]). For example, completion of the M-CHAT at the 30-month WCC visit increased from 67.6 to 82.9% of visits (p < 0.001). Despite the increase in completion rates, the percentage of children identified as “low risk” remained similar (96.6 vs. 96.2%, p = 0.38).

Table 2 Modified Checklist for Autism in Toddlers completion and negative screening rates pre- and post-implementation

	Pre-implementation			Post-implementation
	Number of WCC visits	Completed	Scored low risk	Number of WCC visits	Completed	Scored low risk
18-mo visit	3,132	74.7%	93.7%	2,726	85.5%[a]	92.5%
30-mo visit	3,264	67.6%	96.6%	3,488	82.9%[a]	96.2%

Abbreviation: WCC, well child care.

^a Indicates p < 0.05 compared with pre-implementation.

Patient Health Questionnaire

PHQ-2 completion rates recorded in the EHR increased in all ages from the pre-implementation data (86.4%) to the post-implementation data (93.8%; p < 0.001). The rate of positive screening on the PHQ-2 (defined as a PHQ-2 score of 3 or greater) was higher in the pre-implementation data (11 vs. 7%, p < 0.001). The rate of positive screening among all adolescent and young adult WCC visits was also higher in the pre-implementation period (9.5 vs. 6.7%, p ≤ 0.00; [Table 3]).

Table 3 Patient Health Questionnaire-2 completion and positive screening rates pre- and post-implementation

	Pre-implementation				Post-implementation
Age, y	Number of WCC visits	Completed	PHQ-2 screen positive of PHQ-2 completes	PHQ-2 screen positive of total WCC visits	Number of WCC visits	Completed	PHQ-2 screen positive of PHQ-2 completes	PHQ-2 screen positive of total WCC visits
Total	11,530	86.4%	11.0%	9.5%	28,899	93.8%	7.2%	6.7 %
11–14	6,263	87.1%	10.2%	8.9%	14,666	93.5%	7.5%	6.9%
15–17	3,911	87.4%	12.6%	11.0%	10,111	94.8%	7.4%	7.0%
18+	1,356	80.4%	9.9%	8.0%	4,122	92.4%	5.7%	5.3%

Abbreviations: PHQ-2, Patient Health Questionnaire-2; WCC, well child care.

Screening to Brief Intervention

In the pre-implementation data, 86.7% of all WCC visits had S2BI information recorded (when including incomplete information); in the post-implementation data, it was 93.7% (p < 0.001). The rate of endorsing use of any one of the three substances (tobacco, alcohol, and marijuana) increased from 9.8 to 12.8% (p < 0.001; [Table 4]).

Table 4 Screening to Brief Intervention completion and positive screening rates pre- and post-implementation

	Pre-implementation	Post-implementation
WCC visits, 11+ y	11,530	28,899
Completed screen	86.7%	93.7%
	Response (among completed screens)	Response (among completed screens)
Alcohol
Never	93.2%	89.1%
Once	3.2%	7.8%
Monthly	3.0%	2.5%
Weekly	0.6%	0.6%
Tobacco
Never	96.5%	95.9%
Once	1.8%	3.3%
Weekly or Monthly	1.7%	1.8%
Marijuana
Never	96.0%	95.8%
Once	2.5%	2.7%
Monthly	0.6%	0.8%
Weekly	0.9%	0.7%
Any substance use	9.8%	12.8%

Abbreviations: PHQ-2, Patient Health Questionnaire-2; WCC, well child care.

Asthma Control Test

The total percentage of ACT completed among WCC visits increased from the pre-implementation period to the post-implementation period (2.1 vs. 5.6%, p < 0.001). In pre-implementation, 81.4% of completed ACTs indicated well-controlled asthma in children age 4 to 11 years and 80.4% in children ages 12 years and more (ACT score greater than 19). In post-implementation, well-controlled asthma was more likely to be recorded (83.9% [p < 0.001] and 88.3% [p < 0.001], respectively).

The assignment rules for the ACT were dependent on asthma being present on the problem list rather than staff asking about asthma during the check-in and rooming process, so we investigated prevalence of asthma on the problem list and the frequency with which the ACT was completed within this group. In the pre-implementation data, asthma was less likely to be on the problem list than in the post-implementation data (5.8 vs. 8.9% of WCC visits) and became closer to national averages for asthma prevalence (8.1% of children ages 5–11 and 9.9% of children ages 12–17). In pre-implementation, the number of completed ACTs (as recorded by rooming staff) is larger than the number of patients with asthma (as recorded by clinicians in the problem list) for very young children. In post-implementation, if asthma was added to the problem list by the clinician during the visit, the ACT was not assigned (because the registration process was complete). Thus, only 63% of visits with asthma on the problem list by the end of the visit had completed the ACT during that visit ([Table 5]).

Table 5 ACT completion and control rates pre- and post-implementation

	Pre-implementation				Post-implementation
Age at visit, y	ACT completed of All WCC visits	Well controlled of completed ACTs	WCC visits with asthma on problem list	ACT completed of WCC visits by the total number of patients with asthma on the problem list at WCC	ACT completed of All WCC visits	Well controlled of completed ACTs	WCC visits with asthma on problem list	ACT completed of WCC visits by the total number of patients with asthma on the problem list at WCC
4–7	1.5%	81.4%	1.3%	>100%	2.6%	83.9%	5.2%	49.8%
8–11	2.4%	79.8%	3.7%	63.2%	5.3%	88.3%	8.0%	66.8%
12–15	2.7%	79.2%	9.4%	29.1%	7.6%	88.4%	10.8%	69.9%
16+	2.3%	85.2%	10.2%	22.2%	6.9%	88.3%	11.6%	59.1%
Total	2.1%	36.8%	5.8%	36.8%	5.6%	87.2%	8.9%	63.2%

Abbreviation: WCC, well child care.

Variation by Practice

There was large practice-level variation in questionnaire completion during pre-implementation (M-CHAT = 9–98%, PHQ = 25–98%, S2BI = 25–98%). During post-implementation, there is less variation in completion rates across practices (M-CHAT = 65–96%; PHQ = 79–100%; S2BI = 79–99%). Practices with the lowest completion rates during pre-implementation experienced the greatest improvement in completion rates. Only a small percentage of practices experienced declines in screen completion rates; three practices had >5% decrease in completion rates for the S2BI, and four practices for the M-CHAT and PHQ. ([Fig. 2]).

Fig. 2 Practice-level variation in percent completion of screening questionnaires before and after implementation. Panel A is the Modified Checklist for Autism in Toddlers, panel B is the Patient Health Questionnaire-2, and panel C is the Screening to Brief Intervention.

Workflow Data

Twelve practices were observed. In pre-implementation, an average of 21 patients were observed per practice (range = 8–36) and 45% needed questionnaires. Of these practices, seven did paperwork preparations before patients arrived and five did not. When averaged by practice, 47 seconds were spent on preparing paperwork by front desk staff when a patient arrived. For practices that prepared paperwork before the workday (reporting 5–15 minutes of prep time), the average time was 31 seconds compared with 1 minute and 10 seconds for practices that did not. Once distributed, patients took an average of 3 minutes and 24 seconds to complete and return the paperwork. Three of the practices reported “always” getting paperwork entered into the EHR before a physician saw the patient, seven reported “sometimes,” and two reported “never.” Of the nine practices that did not enter data before the clinician saw the patient, six practices reported the paperwork was in the exam room with the clinician for the visit.

After implementation, an average of 22 patients were observed per practice (range = 7–38) and 43% received questionnaires; tablets were used for 91% of these encounters. It took patients an average of 1 minute and 41 seconds to complete and return the tablets. Of the 96 patients completing questionnaires on tablets, only three were called from the waiting room before completing the questions. In these 12 practices, front desk staff reported that the time spent on tablet management, storage, and charging was usually very brief, but they also reported annoyance at having to clean the tablets between patients (with cleaning wipes) and that sometimes the tablets were nonresponsive or needed to be restarted, and this would substantially disrupt patient flow (n = 4), though this was not observed by the research assistant.

Discussion

This report provides a granular view of the risks and benefits of transitioning to electronic questionnaire capture within pediatric primary care practices. We found that electronic questionnaire administration saved staff time and patient time on average, but also note that front desk staff report significant disruptions when tablets are not working. We report overall improvement in questionnaire completion rates, with notable variation in improvement in completion and in impact on positive screening rates across questionnaires and practices that warrant further discussion.

We identified several lessons learned based on completion rates for each of these questionnaires before and after implementation. First, for the M-CHAT, we note that this questionnaire had already transitioned to electronic implementation, but that it was only completed at two-thirds of WCC visits prior to the broader implementation we study here, with practice-level variation ranging from 10 to 90%. While the M-CHAT process itself was not changed by broader tablet implementation, we saw substantial improvement in M-CHAT completion rates, suggesting a “halo” effect,[19] [20] where increased integration and use of the tablets generally enhanced their use for the M-CHAT as well, perhaps by tablets kept more readily available and/or made a more routine part of the registration process.

The PHQ, S2BI, and ACT were newly transitioned to electronic administration, and each experienced an increase in completed questionnaires recorded in the EHR. Our experience with the ACT offers lessons for questionnaires requiring a diagnosis-specific trigger. First, diagnosis-triggered algorithms will only be as reliable as the EHR information, which in the case of the ACT required a reliable problem list. Both pre- and post-results highlight consequences of inaccurate problem lists, with ACT use exceeding visits with asthma on the problem list during pre-implementation. Second, when problem list documentation becomes more actionable, staff are more likely to ensure that relevant diagnoses are on the problem list, as evidenced by increased documentation of asthma on the problem list post-implementation. While this offers a clear long-term benefit, the ongoing process of updating all problem lists limits the immediacy of the impact of any transition to electronic questionnaires.

Finally, we note that administration of each questionnaire remained imperfect with less than 100% completion. Continued barriers include tablet downtime, new processes being set aside when the office gets particularly busy, patient/parental communication barriers (e.g., vision or literacy), and patient/parents who choose not to complete or become distracted before completing the questionnaires.

We observed differential impact on the rates of positive screening after the transition to electronic administration. For the M-CHAT, the rate of positive screening remained constant, suggesting that children not screened during pre-implementation had a similar symptom rate as those screened. Wide variation at the practice level (likely each with patient populations with similar overall prevalence of symptoms of autism) likely accounts for this pattern and suggests that large groups were not screened rather than selective screening of patients by individual clinicians. While the rate of positive screening remained constant, the overall increase in questionnaire completion rate means that more individuals with symptoms suggestive of autism were identified for early intervention after the transition to electronic questionnaires.

For the PHQ-2, we observed more positive responses during pre-implementation than post-implementation where prior research found higher rates of disclosure using an electronic format.[21] [22] This suggests that there may have been selective screening during pre-implementation of youth with higher apparent symptom burden. During the pre-implementation period electronic capture required administration and manual entry of results, so it is also possible that there was selective entry of data, with a bias toward entering the more concerning results pre-implementation. Additionally, there may be differences in how the questionnaire was presented to teens in the two formats, how often parents complete forms for their children in the two formats, or whether seasonal variation in mood could be contributing.

For the S2BI, despite increased rates of completion, the percentage of responses indicating at least one episode of substance use increased, suggesting greater rates of disclosure by youth through this mechanism, perhaps due to greater comfort disclosing via tablet or more ability to enter without concern about parents viewing the form.[21] [22] Alternatively, clinicians may have hesitated to record positive substance use history in the flowsheets previously due to social or legal implications, resulting in a bias toward more negative results during the pre-implementation period.

The findings in this study of community pediatric practices mirror those in subspecialty pediatric practices that show electronic capture of questionnaires is feasible and effective and did not hinder clinic flow.[23] [24] Limitations of this study include relatively short- time frames pre- and post-implementation comparisons. This study did not evaluate for data entry errors in the pre-implementation period, fully quantify staff workflow, or assess patient, staff, or clinician satisfaction with the tablet implementation. We also cannot exclude that time savings seen in this study were due to the Hawthorne effect.[25] This study's generalizability is limited because it was performed in a single EHR with a well-coordinated EHR oversite group to encourage use of the new data collection procedures. Future work should assess if the improvements seen in the first 3 months post-implementation are sustained over time. Future work should also evaluate changes in clinician actions to questionnaire responses and care pathways to maximize questionnaire use.[26]

Conclusion

Bright Futures and the AAP for preventive pediatric health care recommend 30 different activities including those that use questionnaires for maternal depression screening, autism spectrum disorder screening, and adolescent depression screening. Other questionnaires like the ACT can aid in managing chronic diseases. Clinical staff may be hesitant to use electronic questionnaires because of concerns over adjusting to a new workflow, technical issues, and sanitation concerns about the tablets. The electronic questionnaire implementation system used in this project was designed to reduce paperwork, administrative cost, and data entry errors, as well as improve patient confidentiality. Our findings indicate that electronic data capture can be beneficial to save staff and patients' time and increase questionnaire completion rates, particularly when used for multiple purposes and become a routine part of clinical workflow.

Clinical Relevance Statement

The implementation of electronic questionnaire capture requires an investment of time and finances. This study found that the investing in an electronic questionnaire capture in the waiting rooms of community pediatrics clinics increased questionnaire completion rates while decreasing the time staff spend managing questionnaire administration and data entry.

Multiple Choice Questions

1. Compared with paper-administered questionnaires, electronic questionnaire capture of the PHQ-2 depression screen show:

   • Increased completion rates and decreased positive screen rates

   • Increased completion rates and increased positive screen rates

   • Decreased completion rates and decreased positive screen rates

   • Decreased completion rates and increased positive screen rates

   Correct Answer: The correct answer is option a. The PHQ-2 was expected to be administered at all well child visits for patients age 11 years and older. During paper administration, PHQ-2 completion rates recorded in the EHR was 86.4%. After deployment of the electronic capture system, the completion rates recorded in the EHR increased to 93.8% (p < 0.001). However, the rate of positive screening for depression was lower after deployment of the electronic capture system both among patients who completed the questionnaire (11 vs. 7%, p < 0.001) and among all appropriate well child visits (9.5 vs. 6.7%, p ≤ 0.001). It is unknown if this reflects selective screening in the paper administration era, selective data entry in the paper administration era, or differences in questionnaire responses on paper compared with tablet.

2. After implementation of electronic questionnaire capture, variation in the questionnaire completion rates across practices:

   • Increased

   • Stayed the same

   • Decreased

   Correct Answer: The correct answer is option c. There was significant practice-level variation across the 45 practices in questionnaire completion during the paper administration period (9–98% for the M-CHAT; 25–98% for the PHQ-2; 25–98% for the S2BI). After the deployment of electronic questionnaire capture, there was less variation in questionnaire completion rates across practices (65–96% for the M-CHAT; 79–100% for the PHQ-2; 79–99% for the S2BI).

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

This project was approved by the UPMC Quality Improvement Review Committee (1629); projects approved by this committee do not meet the definition of human subjects research and therefore do not require institutional review board approval.

• References

• 1 American Academy of Pediatrics. Bright Futures/AAP recommendations for preventive pediatric health care (Periodicity Schedule) Updated: March, 2019. Accessed January 2020 at: https://www.aap.org/en-us/documents/periodicity_schedule.pdf
• 2 Centers for Medicare & Medicaid Services. Core set of children's health care quality measures for Medicaid and CHIP (Child Core Set). 2019 ; Accessed January 2020 at: https://www.medicaid.gov/medicaid/quality-of-care/downloads/medicaid-and-chip-child-core-set-manual.pdf
• 3 Dinakar C, Chipps BE. Section on Allergy and Immunology, Section on Pediatric Pulmonology and Sleep Medicine. Clinical tools to assess asthma control in children. Pediatrics 2017; 139 (01) e20163438
• 4 Van Cleave J, Kuhlthau KA, Bloom S. et al. Interventions to improve screening and follow-up in primary care: a systematic review of the evidence. Acad Pediatr 2012; 12 (04) 269-282
• 5 Hirai AH, Kogan MD, Kandasamy V, Reuland C, Bethell C. Prevalence and variation of developmental screening and surveillance in early childhood. JAMA Pediatr 2018; 172 (09) 857-866
• 6 U.S. Department of Health and Human Services, Health Resources and Services Administration, Maternal and Child Health Bureau. Child health USA 2014. Rockville, Maryland: U.S. Department of Health and Human Services; 2015 ; Accessed January 2020 at: http://mchb.hrsa.gov/chusa14
• 7 Snyder CF, Aaronson NK, Choucair AK. et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res 2012; 21 (08) 1305-1314
• 8 Foster A, Croot L, Brazier J, Harris J, O'Cathain A. The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient Rep Outcomes 2018; 2 (01) 46
• 9 Haverman L, Engelen V, van Rossum MA, Heymans HS, Grootenhuis MA. Monitoring health-related quality of life in paediatric practice: development of an innovative web-based application. BMC Pediatr 2011; 11 (01) 3
• 10 Schepers SA, Sint Nicolaas SM, Haverman L. et al. Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care. Psychooncology 2017; 26 (07) 951-959
• 11 Schepers SA, Sint Nicolaas SM, Maurice-Stam H. et al. First experience with electronic feedback of the Psychosocial Assessment Tool in pediatric cancer care. Support Care Cancer 2017; 25 (10) 3113-3121
• 12 Schepers SA, Haverman L, Zadeh S, Grootenhuis MA, Wiener L. Healthcare professionals' preferences and perceived barriers for routine assessment of patient-reported outcomes in pediatric oncology practice: moving toward international processes of change. Pediatr Blood Cancer 2016; 63 (12) 2181-2188
• 13 Barthel D, Fischer KI, Nolte S. et al. Implementation of the Kids-CAT in clinical settings: a newly developed computer-adaptive test to facilitate the assessment of patient-reported outcomes of children and adolescents in clinical practice in Germany. Qual Life Res 2016; 25 (03) 585-594
• 14 Robins DL, Casagrande K, Barton M, Chen CM, Dumont-Mathieu T, Fein D. Validation of the modified checklist for Autism in toddlers, revised with follow-up (M-CHAT-R/F). Pediatrics 2014; 133 (01) 37-45
• 15 Levy S, Weiss R, Sherritt L. et al. An electronic screen for triaging adolescent substance use by risk levels. JAMA Pediatr 2014; 168 (09) 822-828
• 16 Schatz M, Sorkness CA, Li JT. et al. Asthma Control Test: reliability, validity, and responsiveness in patients not previously followed by asthma specialists. J Allergy Clin Immunol 2006; 117 (03) 549-556
• 17 Nathan RA, Sorkness CA, Kosinski M. et al. Development of the asthma control test: a survey for assessing asthma control. J Allergy Clin Immunol 2004; 113 (01) 59-65
• 18 Jia CE, Zhang HP, Lv Y. et al. The Asthma Control Test and Asthma Control Questionnaire for assessing asthma control: Systematic review and meta-analysis. J Allergy Clin Immunol 2013; 131 (03) 695-703
• 19 Utter GH, Maier RV, Rivara FP, Nathens AB. Outcomes after ruptured abdominal aortic aneurysms: the “halo effect” of trauma center designation. J Am Coll Surg 2006; 203 (04) 498-505
• 20 Marrie TJ. The halo effect of adherence to guidelines extends to patients with severe community-acquired pneumonia requiring admission to an intensive care unit. Clin Infect Dis 2005; 41 (12) 1717-1719
• 21 Gnambs T, Kaspar K. Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis. Behav Res Methods 2015; 47 (04) 1237-1259
• 22 Bradford S, Rickwood D. Psychosocial assessments for young people: a systematic review examining acceptability, disclosure and engagement, and predictive utility. Adolesc Health Med Ther 2012; 3: 111-125
• 23 Iturralde E, Adams RN, Barley RC. et al. Implementation of depression screening and global health assessment in pediatric subspecialty clinics. J Adolesc Health 2017; 61 (05) 591-598
• 24 Anderson LM, Papadakis JL, Vesco AT. et al. Patient-reported and parent proxy-reported outcomes in pediatric medical specialty clinical settings: a systematic review of implementation. J Pediatr Psychol 2020; 45 (03) 247-265
• 25 McCambridge J, Witton J, Elbourne DR. Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol 2014; 67 (03) 267-277
• 26 Jensen RE, Chan KS, Weiner JP, Fowles JB, Neale SM. Implementing electronic health record-based quality measures for developmental screening. Pediatrics 2009; 124 (04) e648-e654

Publication History

Received: 17 November 2020

Accepted: 26 February 2021

Article published online:
14 April 2021