Implementing an Electronic Patient-Reported Outcome and Decision Support Tool in Early Intervention

Abstract

Objective The aim of the study is to identify and prioritize early intervention (EI) stakeholders' perspectives of supports and barriers to implementing the Young Children's Participation and Environment Measure (YC-PEM), an electronic patient-reported outcome (e-PRO) tool, for scaling its implementation across multiple local and state EI programs.

Methods An explanatory sequential (quan > QUAL) mixed-methods study was conducted with EI families (n = 6), service coordinators (n = 9), and program leadership (n = 7). Semi-structured interviews and focus groups were used to share select quantitative pragmatic trial results (e.g., percentages for perceived helpfulness of implementation strategies) and elicit stakeholder perspectives to contextualize these results. Three study staff deductively coded transcripts to constructs in the Consolidated Framework for Implementation Research (CFIR). Data within CFIR constructs were inductively analyzed to generate themes that were rated by national early childhood advisors for their relevance to longer term implementation.

Results All three stakeholder groups (i.e., families, service coordinators, program leadership) identified thematic supports and barriers across multiple constructs within each of four CFIR domains: (1) Six themes for “intervention characteristics,” (2) Six themes for “process,” (3) three themes for “inner setting,” and (4) four themes for “outer setting.” For example, all stakeholder groups described the value of the YC-PEM e-PRO in forging connections and eliciting meaningful information about family priorities for efficient service plan development (“intervention characteristics”). Stakeholders prioritized reaching families with diverse linguistic preferences and user navigation needs, further tailoring its interface with automated data capture and exchange processes (“process”); and fostering a positive implementation climate (“inner setting”). Service coordinators and program leadership further articulated the value of YC-PEM e-PRO results for improving EI access (“outer setting”).

Conclusion Results demonstrate the YC-PEM e-PRO is an evidence-based intervention that is viable for implementation. Optimizations to its interface are needed before undertaking hybrid type-2 and 3 multisite trials to test these implementation strategies across state and local EI programs with electronic data capture capabilities and diverse levels of organizational readiness and resources for implementation.

Keywords

early intervention - assessment - goal setting - implementation strategies - family engagement - facilitators and barriers - children - patient-reported outcomes

Background and Significance

Part C of the Individuals with Disabilities Education Act[1] authorizes states to create early intervention (EI) programs, which currently serve 3.7% of U.S. children ages 0 to 3 years old.[2] EI programs aim to offer family-centered services in the child's natural environment to improve child and family outcomes, such as increasing family access to valued activities for their child to participate and develop skills.[3] However, children and families experience disparities in their access to and use of EI services that have historically relied on face-to-face processes of care.[4] [5] [6] Practice-based strategies for improving equitable EI processes, such as those proposed for EI referral and evaluation,[7] hold promise for improving EI service quality.

The use of patient-reported outcomes (PROs) is endorsed as a practice-based strategy for advancing value-based services for children and families.[8] Electronic patient-reported outcomes (e-PROs) can serve as an informatics tool for including the child and family perspective regarding their needs and priorities when designing and evaluating services.[9] [10] In EI, e-PROs can be used in direct service provision to consider family needs and priorities for tailoring the development of the individualized family service plan (IFSP) that should include functional goals and strategies for goal attainment based on family-identified priorities and inclusive of their expertise. Some e-PROs can also be useful for tracking progress to guide shared and data-driven decisions about tailoring interventions with families.[10] In both cases, e-PROs support equitable services via quantifiable estimates from the family that are consistent in the information they capture and minimize the need for clinician interpretation (and implicit bias).[10] Beyond their benefits to individual-level services, e-PROs yield data that can be aggregated for conducting robust health services and implementation research, to demonstrate the value of EI services and guide decisions for improving these services.[10] [11] While there are challenges to appraising and implementing e-PROs in pediatric contexts like EI,[11] [12] [13] [14] [15] they hold promise for accelerating quality improvement of these services in programs with electronic data capture systems.

The Young Children's Participation and Environment Measure (YC-PEM) is a promising e-PRO informatics tool that offers an evidence-based strategy to deliver family-centered EI in programs with electronic data capture systems.[16] [17] [18] [19] [20] [21] [22] [23] [24] Its completion takes approximately 20 to 30 minutes[18] and yields a summary report for caregivers to share with their EI service providers.[25] EI stakeholders have participated in research to develop, evaluate, and implement it per best practice standards.[26] [27] [28] Specifically, the YC-PEM e-PRO and its companion goal setting application were developed primarily in partnership with EI providers[17] [29] and EI families,[18] [23] [30] [31] [32] and EI program leadership (PL) partnered in developing organizational infrastructure needed to incentivize EI provider involvement in research on this topic.[33] Together, these EI stakeholders informed decisions about how and when to introduce the YC-PEM e-PRO option for inclusive service design toward meaningful EI outcomes, including their child's participation in valued activities.[34] [35] For example, these EI stakeholders have shaped decisions to explore implementing the YC-PEM e-PRO as part of the child's service visit and annual evaluation of progress.[17] [18] [36]

To date, EI stakeholders representing differing perspectives about information exchange during family-provider encounters[37] and at all levels of an EI organization (i.e., families, EI providers and service coordinators (SC), and PL) have been engaged to develop the YC-PEM e-PRO option (e.g., layout of the summary report) and select implementation strategies (e.g., research group infrastructure, timing of its implementation). However, they have not yet been engaged to identify the full range of relevant supports and barriers to implementing the YC-PEM e-PRO option across EI programs, for improving both individual-level information exchanges within routine EI workflows and for guiding decisions for quality improvement of EI services.[38] [39]

Objectives

This study is part of the latest phase of YC-PEM e-PRO implementation research,[40] with two aims: (1) to identify facilitators and barriers for implementing the YC-PEM e-PRO option in an EI program, from the perspectives of families in EI, SC (i.e., facilitate EI access and coordinate services per IFSPs), and PL (i.e., oversee EI program's infrastructure and functions; aim 1); and (2) to examine the relevance of supports and strategies for scaling its implementation across multiple state and local EI programs (Aim 2). The implementation outcomes captured in this study will guide decisions about YC-PEM e-PRO scalability across EI programs with electronic data capture systems.

Methods

Study Design and Setting

This cross-sectional study employed an explanatory-sequential (quan> QUAL) mixed-methods study design.[41] This study is part of a larger implementation research project that follows a hybrid type-1 effectiveness-implementation research approach.[42] [43] A mixed-methods study design is appropriate when one research approach is insufficient for addressing the main research problem under investigation.[41] The choice to mix quantitative and qualitative methods during data collection was deemed most appropriate based on the breadth and nature of stakeholder input needed. Choices for sequencing and emphasizing qualitative data collection were made a priori by the corresponding author who had prior experience in using this particular mixed-methods study design (M.A.K.),[29] [30] to further explain a subset of quantitative findings concerned with pilot implementation of the intervention.[41] This research took place at a large, urban, and non-university affiliated EI program in the Denver Metro catchment of Colorado. Ethical approval was obtained by the Institutional Review Board at the University of Illinois (protocol #2020–0555) and the University of Colorado (protocol #20–2380). We registered the Parent-Reported Outcomes to Strengthen Partnership within the Early Intervention Care Team (PROSPECT) trial at clinicaltrials.gov (NCT04562038) and published protocol details.[40]

Participants

During the implementation trial period, three groups of stakeholders were recruited from a large, cosmopolitan EI program: Families, SC, and PL. This EI program was comprised of more than 45 EI service providers (e.g., SC, skilled therapists, special educators) serving over 2,200 children and families. Families varied on sociodemographic factors (e.g., 51.7% with Medicaid and 46.5% as Non-Hispanic White) (Jodi Litfin-Dooling, personal communication, June 28, 2022): Six families were assigned to the intervention group during the PROSPECT trial and indicated interest in participating in a follow-up interview or focus group. Nine SC were enrolled, including five SC who were engaged in conducting the larger trial and four SC who were not involved in the trial. PL (n = 7) comprised of directors, supervisors, and data and program managers. SC and PL stakeholders with at least 2 years of EI experience were recruited by email. Families were included if they: (1) were at least 18 years old; (2) identified as a parent or legal guardian of a child receiving EI; (3) were able to read, write, and speak English; (4) had internet and telephone access; and (5) had a child 0 to 3 years old who received EI for 3 or more months. Each participant was issued a $20 electronic gift card.

Data Collection and Procedures

SC and PL stakeholders completed a demographic questionnaire online and prior to qualitative data collection. Quantitative data on implementation were collected as part of a larger pragmatic trial on the effectiveness of the YC-PEM e-PRO, when paired with a co-designed, program-specific decision-support guide, on select outcomes.[40] These quantitative data on implementation were summarized and used to prompt stakeholders to explain findings during qualitative data collection.[40] Specific data on sample characteristics and implementation included: (1) items administered to caregivers online (e.g., “How helpful was the [instructional] video for completing the YC-PEM e-PRO questionnaire?” [from 1 = not helpful to 4 = very helpful]); and (2) items administered to SC online during their monthly meetings (e.g., “How helpful were monthly meetings with other SC in the intervention group” [from 1 = not helpful to 4 = very helpful]; “How helpful were the YC-PEM e-PRO results within your IFSP documentation to guide your conversation with parents during the annual evaluation of progress” [from 1 = not helpful to 4 = very helpful]).

The qualitative data collection phase included six families each taking part in a 30 to 60-minute semi-structured virtual interview, and two in-person focus groups (3–4 SC and PL per group) at an off-site location ([Supplementary Appendix A], available in the online version). Additionally, one SC was rescheduled for a virtual interview. Individual interviews were chosen for families to gain deeper insight into the study topic, and focus groups for SC and PL were used to encourage discussion by employees of the organization.[44] Sessions were co-facilitated by two of four study staff (V.C.K., J.G.C.S., N.J.M., and M.A.K.), most of whom had prior qualitative and/or mixed-methods research experience for examining mechanisms of EI care coordination and/or further developing the PEM approach for use in EI. These study staff used semi-structured interview guides, one version of which was piloted with a caregiver of a child who had received EI services. During each session, participants (families, SC, and PL) were presented with data displays (bar graphs and tables) of quantitative findings and then asked to explain the results shown. We also asked all stakeholders to elaborate or explain facilitators and barriers to implementation that had surfaced during monthly SC meetings (e.g., optimal timing of YC-PEM e-PRO completion for use at the IFSP meeting). In-person focus groups and virtual interviews were audio and video recorded.

Data Analysis

Quantitative findings were analyzed descriptively and presented as percentages to all stakeholders during the qualitative data collection phase. Each completed session yielded a recording that was transcribed verbatim by a staff member (J.G.C.S.) and imported into NVivo 13.0[45] for analyses. Study staff performed content analyses on these transcripts, listening to audio and viewing video recordings as needed for coding accuracy.

For Aim 1, two of three study staff (V.C.K., J.G.C.S., S.R.) concurrently analyzed each of three sets of transcripts for a stakeholder group (i.e., families, EI service providers, and PL). Study staff first used deductive content analysis[46] to independently code data to established constructs for four of five domains in the Consolidated Framework for Implementation Research analytic framework (CFIR),[47] reaching a Kappa of 0.86 across all stakeholders and CFIR domains. The CFIR is an established determinant framework that is appropriate for examining multiple levels of influence within the organizational context.[48] The CFIR domain “characteristics of individuals” was excluded from the codebook because this study involved an established High Value EI Research Group with known characteristics. Coding discrepancies were resolved through discussion to achieve consensus on final constructs, combining select CFIR constructs with similar coded content for ease of interpretation. The coded data for each CFIR construct (original or combined) were then inductively analyzed to generate categories that were further collapsed into themes within original or combined CFIR constructs. These themes were presented using frequency counts per stakeholder group, and deidentified excerpts from stakeholders' full transcripts to demonstrate frequency, extensiveness, and specificity of concepts or themes.[49]

For aim 2, we shared created themes specific to SC and PL perspectives with six national early childhood advisory group members to check their relevance across multiple EI programs. These advisors have professional histories including EI practice experience and contribute to improving EI quality in multiple states, but they have limited knowledge of the YC-PEM e-PRO that has been previously designed and tested in close partnership with caregivers of young children with developmental need.[18] [23] [30] [31] [32] Therefore, geographically diverse professional expertise was prioritized in this study phase to help identify scalable supports and strategies given the state-to-state variability in EI service provision. During a 30-minute virtual session, two staff (S.R. and V.C.K.) co-facilitated with advisory leadership (A.S., L.J., and K.B.), to solicit advisors' ratings on main findings according to: (1) the importance of the support, strategy, or barrier for implementation and (2) how easy this strategy, support, or barrier would be to achieve during implementation. Each item response was assigned a weighted percentage for importance and difficulty (i.e., 25% = not important at all/very difficult to 100% = very important/very easy) and multiplied by the number of respondents who indicated each response option. Item responses were used to confirm thematic findings (e.g., each main thematic finding was perceived to be important to one or more advisors) and informed the ranking of the CFIR constructs within CFIR domains. SC and PL perspectives were centered in this study because caregiver perspectives were emphasized in prior phases of implementation research on this topic, and because of this study's focus on systemic implementation.[18]

Results

As shown in [Table 1], participants were six family caregivers (four female, two male) of children receiving EI services, nine female SC, and seven female PL. Families differed according to their child's developmental need, but each had completed graduate coursework or a degree, and most families earned above the U.S. median household income.[50] Most families (83.3%) perceived technology to be very important for working with their EI team. SC and PL had 2 to 15 years of EI experience and diverse fields of degree.

Table 1 Early intervention stakeholder characteristics

Stakeholder	Caregiver race	Caregiver role	Family annual income ($)	Child diagnosis/delay
F1	Caucasian	Mother	>100,000	Bilateral hearing loss
F2	Caucasian	Father	90,001–100,000	Congenital diaphragmatic hernia, tracheostomy, gastrojejunostomy tube, global delays
F3	Caucasian	Father	40,001–50,000	Developmental delay (no diagnosis)
F4	Caucasian	Mother	>100,000	Childhood apraxia of speech
F5	Caucasian	Mother	>100,000	Hearing loss
F6	Black or African American	Mother	70,001–80,000	Developmental delay (no diagnosis)
Stakeholder	Years of service in organization	Years of EI experience	Field of degree
SC1	2.5	2.5	Human development Early childhood education
SC2	2	2	Early childhood education
SC3	2	2	Speech and language pathology
SC4	16	14	Social work
SC5	2	2	Not disclosed
SC6	2	2	Social work
SC7	8	8	Human development Family studies
SC8	4	4	Education Human development
SC9	11	15	Sociology
Stakeholder	Years of service in organization	Years of EI experience	Field of degree
PL1	3	3	Sociology Women's studies
PL2	12	12	Psychology
PL3	2.5	2.5	Social work
PL4	2	15	Speech and language pathology
PL5	12	14	Social Work
PL6	4	4	Education
PL7	5	7	Social Work Early childhood education

Abbreviation: EI, early intervention; F, family; PL, program leadership; SC, service coordinator.

As part of Aim 1, all three stakeholders (i.e., families, SC, and PL) identified supports and barriers across multiple constructs for each of four CFIR domains ([Fig. 1]). We report descriptive quantitative results using percentages and frequency counts (n) and exemplar quotes for 19 themes associated with original or combined CFIR constructs ([Table 2]). Findings within each CFIR domain are presented in rank order based on advisory feedback obtained as part of Aim 2.

Fig. 1 Qualitative findings by consolidated framework for implementation research (CFIR)[47] domains and constructs. EI, early intervention; e-PRO, electronic patient-reported outcomes; IFSP, individualized family service plan; YC-PEM, Young Children's Participation and Environment Measure.

Table 2 Consolidated framework for implementation research (CFIR)[47] domains and constructs with exemplar quotes

CFIR domain	CFIR construct as addressed by EI stakeholder group(s)	Exemplar quotes	Construct importance and ease (%)
Intervention characteristics	Complexity, design quality and packaging	Data-driven programmatic decision-making “…I mean that [aggregate of what the families are needing] could help with us thinking even about hiring and staffing and training […] to improve the quality and accessibility of our intervention services and [what things] were really prevalent in our community as needs. And we could think about like, ‘Oh, is there a resource out there that we can engage and help our families to know about?’.” (PL2) Easy to complete with adequate resources “It was pretty straightforward to me, but I think, in my case, I was willing to devote time to it. It was just that, the, the day that I did it turn into a really busy day, and then I was just trying to catch up to the rest of the day. Um, but it really just came down to the fact that the survey let me do that, it let me keep going. Right, moving quickly.” (F3)	80%
	Adaptability and trialability	Starting small with EI provider engagement “I think we were definitely taking more of a thoughtful approach [by starting with a smaller group]. We're kind of finding what's working, what's not working, and tweaking things […] with the intention of also adding more people, adding their input and sort of, you know, continue to grow it from there.” (SC7) “I think having people that have already been more familiar with it and then slowly integrating into the rest would be helpful. Because if you, do it all at once, there's going to be a lot of questions and a lot of things that might not go smoothly. But this way, like the smaller group is more controlled, and […] as you add people, there's people that can ask or answer those questions and help fix anything” (SC6) Diversifying by language and mode of administration “I'm wondering if .. if there was a button that somebody, you know, prerecorded if the family needed help or they didn't quite understand […]. Kind of giving them like a face with the voice. And I also think about families with different learning styles. Um, if it's all reading that might not be as accessible […]. So, if we had like a text to speech option on it, and some more of those example piece. I think that would be beneficial” (PL6)	75%
	Relative advantage	Capturing family priorities and doing so, ahead of time “It's a lot of extemporaneous [information] during the meeting […] and so I think it was […] helpful to remember like to think about sort of what were the things we wanted from the service that we were getting [to be] better prepared” (F2) “[It] is an important tool because it helps us to say, here are the things that you want to focus on […] for your plan.” (SC8) “I think the YC-PEM tool is unique to help the family get started and thinking ahead of time […] ‘How have my routines changed in the last six months?’ and ‘What might I want to work on?’ Instead of just at the end of the meeting being like, ‘So [we] have this goal.’ […] I think it's a really great opportunity for families to get to have some reflection outside of that meeting, without people staring at them or feeling pressured to answer questions. Which just makes the meeting in my opinion just more authentic and […] very genuine.” (PL5) Gaining efficiency for IFSP development “[…] the question that was asked which was very narrow and specific and so there was a ton of questions about a few very specific activities that he's never been involved in and so kind of felt like that made the whole thing look like he wasn't really involved in life at all in ways that we didn't feel like were representative of our experience.” (F2)	73.5%
	Intervention source	N/A (no data mapped to this construct)
	Evidence strength and quality	N/A (no data mapped to this construct)
	Cost	N/A (no data mapped to this construct)
Process	Engaging	Affluence and linguistic diversity “[This was] made available for families who […] self-select if they speak English. And so when I'm looking at this I'm also thinking about how families, Hispanic, LatinX, or Spanish origin, if they're like, ‘Oh my goodness, this is going to be fully in English’. They might not feel comfortable enough to proceed.” (PL5) Provider engagement and identity “[The] podcast was such a great motivator [...]and so I think that speaks volumes. When we had talked about the podcast and then that great feedback the family gave us about feeling more prepared for their meeting […] I think it spoke to [the new hire] and now that she [is part of the project and] is going through grad school she wants to be better connected to research. And so this is fabulous, really.” (PL1) “I love the way that we also divided up the work and that we had [a family engagement specialist] there who was our outreach specialist and additional layer of support because I could go to her and bounce ideas off to make sure that our staff understood what those next steps were. And I think also the level of communication [and accessibility of team members] has been really valuable as well, because when I did run into a stumbling block, I'm like' this is going to impact what we're doing', I could go to call [the project coordinator], and we could brainstorm. I didn't need a scheduled meeting with [the project coordinator] every month. I just knew I needed to know that I can reach out yeah exactly.” (PL1)	80%
	Reflecting and evaluating	Protected space for provider reflection and learning “I definitely think at first it would be helpful to have like, like the monthly meetings that we had or like biweekly […] like even if I felt like I didn't always have questions going into the meeting, I definitely felt like it was always helpful to hear what other people had to say, and just kind of have that time set aside to be able to […] think about it and talk about it.” (SC5) Instructional video for family reflection “Video probably captures the widest audience and that, you know, allows for those who […] vary in literacy or ability to read through the material to have a clear video. So I think from that perspective, having a video like that is probably still the best option. There may be some who prefer to take that in other ways like reading, or simply listening. But I think to be honest for the, the audience that this likely targets that's probably the high yield mechanism.” (F4) “I feel like I do need the visual and if it's in a video it's definitely more helpful and easier to understand than just, you know, a questionnaire, or it being written. So I feel like this is pretty accurate to how my families would react to it.” (SC6) “[…]but also maybe for parents who are more often, or more intensely distracted by their kids. And, and have a hard time just sitting down and reading something to have something playing that they can listen to can be helpful.” (SC8)	71.3%
	Planning and executing	Timing and automating of processes “it may not have actually been that far that long time, but it felt like it was longer than it should be. To really remember the answers.” (F3) “a month could be huge in development, the life of a child […] your kids like starts daycare something you know like their lives are completely different.” (SC3) “if I knew that this was serving the purpose of informing decision making for like what our services are going to look like, then I think it would make it more important. I guess like more just meaningful.” (F5) Optional or required “I also feel like fine about it being like an optional thing because it's like the parents that have the time and want to do all this extra stuff to really get the most out of this experience, like good for them, and then for the parents who like, it's like an accomplishment for them to just show up to their therapy session […] I feel like I go back and forth between like maybe the people that it would help the most would be the people that would be less likely to volunteer.” (SC5)	68.3%
Inner setting	Culture, network and communication	Education and practice for clear communication “… in the past, I could have looked at some parents differently than others, because I thought some parents were going to come to the meeting prepared and some weren't and it's really just the biases or deficit thinking could get in the way.” (F5) “I needed somebody to translate it to me, yeah, but I do feel like when we were then implementing like you gave me the YC-PEM and I could see it and I walked through it as though I was the parent, … boosted the confidence of our staff to do it and myself.” (PL1)	92.5%
	Structural characteristics and Implementation Climate	Increasing and changing workload demands “… as a service coordinator, it does take that extra step to prepare and make sure that we're reading that. And there's sometimes, as I'm sure that N and A can attest to, where you just don't have that extra time. So having that extra step could make it more difficult to make sure the tools being more of a help, than it is not.” (SC1) “And I have to say that at this point, for example, I'm handling 110 cases. It wouldn't be feasible for me to add another layer. I'm not discarding that it's just at this moment it wouldn't be practical and feasible, at least from my perspective. I already, on top of the parents [I am] like, ‘oh please sign this document, please’.” (SC2) “[if you] say that that research is important, then you're just going to implement it as part of your workflow. And you're going to demonstrate it in active ways to your staff by having them be a part of it or at least dip their toe and try it.” (PL1) Incentivizing organizational change “if I'm being 100% honest, if I had not been offered an incentive and looked at that survey and it was optional, I probably would not have taken the time to fill it out.” (F1) “some incentive. And, like, probably stronger, more like people who've been there for a while, I guess.” (SC3)	76.3%
Outer setting	Cosmopolitanism	Improving family-centered pathways into EI “I want a whole team of navigators, and like every touch point that we have, [so] we're getting [in] there, like coming in at the connection from referral to evaluation […] and the Denver Health Navigator now coming in, when the pediatrician is doing the screening to help you explain to the family like what EI is [and] understanding the model, I think at our touch points.” (PL4) “The Head Start [is] for kids that are over three so it's the Early Head Start programs. They're in the same facilities [and] we work really closely with [them]. […] If we were able to […] market to gynecologist, right, to OB-GYNs […]. She [the gynecologist] didn't have a clue that [EI] was even available […]. So I think there are so many places that we can try to […] get our, our message out there where we can start when they're pregnant. Because, yeah, I was blown away that [my gynecologist] did not have any idea what early intervention was.” (PL5) Improving family-centered pathway within and across EI programs statewide “So I'm really excited to see like as they [the state] continue to revise this tool, and if we can use it for those ARs [annual reviews…] I would want to keep an eye on that […] as like an additional quality tool [to] see, like are we writing better plans.” (PL1)	70%
	Patient needs and resources	Meeting needs of families accessing EI “Part of the point of early intervention is like meeting the family where they're at […]. Sometimes they can't even afford rent and that's what they're thinking about is like […] we always talked about like the Maslow's hierarchy of needs” (SC5) “I think it really shows that issue of access. There […] are barriers to EI but it's available to everybody. […] seeing who engages, who enrolls and who was served is a reflection of kind of the issues that might be out there […]. Non-White and lower income families were not making it into early intervention like at the same rate […]. This is very consistent with that data.” (PL6)	65%
	External policies and incentives	COVID Restrictions Impacting YC-PEM e-PRO Completion and Family Enrollment “There are a lot of questions about how do you participate in activities in the community…He doesn't because we've been locked down for 15 months […] as a family have been particularly cautious because I'm lucky enough to work from home and my wife wasn't working.” (F3) “We're really focusing on trying to get families […] to actively engage in services when the pandemic first hit and we had to go completely telehealth.[…] if you're going back to just trying to meet your basic needs, you're not going to feel capable to engage.” (PL1) “And they're doing what they can to be getting services for their child […] it's very stressful especially now with COVID still happening and everything. I feel like parents are feeling more isolated just […] because they have been in their house for so long and they don't feel comfortable with their kids going to daycare maybe or they've lost their job because of everything that's going on […]. It's a lot of outside factors that they just can't […] add one more thing to their plate.” (SC6) “That's kind of the time we're in right now […]. People haven't had a lot of interactions with adults, and you ask them one question and they fill out their entire life story […] because you haven't seen adults, like face to face, for quite some time […] some people need that interaction.” (SC6)	50%
	Peer pressure	N/A (no data mapped to this construct)	N/A

Abbreviations: CFIR, consolidated framework for implementation research; EI, early intervention; e-PRO, electronic patient-reported outcome; F, family; IFSP, individualized family service plan; PL, program leadership; SC, service coordinator; YC-PEM, Young Children's Participation and Environment Measure.

CFIR Domain: Intervention Characteristics

EI stakeholders discussed two themes in each of three original and combined constructs: (1) “complexity, design quality, and packaging,” (2) “adaptability and trialability,” and (3) “relative advantage” ([Fig. 1]).

CFIR Constructs: Complexity, Design Quality, and Packaging

Data-Driven Programmatic Decision-Making

PL (n = 2) recognized the potential of the YC-PEM e-PRO to inform data-driven, programmatic decision-making by being “really helpful for us [EI program] to just have a good sense in aggregate of what the families were needing help with” (PL2). PL further described how this aggregated information could help them improve quality and accessibility of their services.

Easy to Complete with Adequate Resources

All stakeholder groups (families [n = 2], SC [n = 1], PL [n = 1]) found the YC-PEM e-PRO to be clear and easy to navigate. Some families (n = 3) and SC (n = 2) recognized potential challenges to YC-PEM e-PRO completion by families without “active internet access […], intact vision and hearing” (F2), English proficiency, and/or adequate time. SC also reflected on time pressure, sharing that SC could become overwhelmed when integrating information from a YC-PEM e-PRO report during their IFSP meeting and wanting to “make sure that you're kind of hitting all those points that the family, kind of wanted to improve and work on” (SC3). They suggested increasing use of skip logic and/or administering it in modules.

CFIR Constructs: Adaptability and Trialability

Starting Small with EI Provider Engagement

All SC (100%) in the intervention group perceived the benefit to starting with a smaller provider group when adapting and testing the implementation of the YC-PEM e-PRO in their EI program. In explaining this result, they described being able to engage in trial and error and access peer support by creating “that kind of community […] to bounce off of each other if we need support” (SC8), collectively and collaboratively brainstorming ideas.

Diversifying by Language and Mode of Administration

All three stakeholder groups (i.e., families [n = 6], SC [n = 4], and PL [n = 4]) further described the benefit to offering the YC-PEM e-PRO in different languages and expanding modes of administration, suggesting online or phone support (“complete a quick call with them” [SC1]) and the option to “click on the audio [and have someone] read the question [to] you” (F6).

CFIR Construct: Relative Advantage

Capturing Family Priorities and Doing So, Ahead of Time

All SC perceived the YC-PEM e-PRO as very helpful (67%) or helpful (33%) in representing family priorities for goal setting. These results reveal the YC-PEM e-PRO offers performance advantages, which were further explained by most stakeholders (i.e., families [n = 5], SC [n = 7], and PL [n = 7]) as helping families to reflect on their needs and priorities prior to the IFSP meeting, to strengthen meaningful discussions and create quality goals for the service plan during the meeting.

In addition to relieving families of conceptualizing and articulating their priorities on the spot at the IFSP meeting, the YC-PEM e-PRO option afforded SC “to see what's in the environment [and] read for what the family might want or need (SC1)” due to the coronavirus disease (COVID) restrictions that hindered their access to this information via home visits.

Gaining Efficiency for IFSP Development

SC reported that the YC-PEM e-PRO is helpful (67%) or somewhat helpful (33%) to guide the family-professional exchanges during IFSP development. In explaining these results, some families recognized that information generated by the YC-PEM e-PRO underlies its benefit to IFSP development, depending on family expectations (e.g., information is less helpful for those activities that are less valued by families [n = 2] or not deemed to be problematic owing to COVID restrictions [n = 3]).

Several SC (n = 3) shared how YC-PEM e-PRO results help them to facilitate more efficient IFSP meetings, “because you're not going back and asking them to repeat information” (SC1) and, in turn, can free up resources so that “we can do more for families utilizing it” (SC6).

CFIR Domain: Process

For this domain, EI stakeholders described two themes for each of these original and combined constructs: (1) “engaging,” (2) “reflecting and evaluating,” and (3) “planning and executing” ([Fig. 1]).

CFIR Construct: Engaging

Affluence and Linguistic Diversity

When reflecting on participant engagement, one member of each stakeholder group mentioned our inclusion criteria, with respect to “a selection bias process […] the people who answered the survey are the ones who didn't have barriers that stop them from answering” (F2). Most PL (n = 5) explained the value of expanding engagement to Latinx families and those with undocumented status and who may not identify as English speaking. Similarly, two PL mentioned a tendency of SC with higher education levels to volunteer as early adopters and implementers of the YC-PEM e-PRO.

Provider Engagement and Identity

SC and PL (n = 4) suggested strategies for engaging providers as early adopters. Strategies ranged from sharing information (e.g., family quotes, their podcast on the history of this research partnership)[51] to teamwork (i.e., designating project coordinators at research and EI sites, designating EI staff to screen and recruit families, recruiting a family engagement specialist, and organizing team experiences to train and troubleshoot). Several PL (n = 3) valued their transition to including all new hires to their “High Value EI Research Group” as part of their onboarding process and recognized their need to avoid calling it a “writing group,” a misleading informal group name risking to “keep them [SC] from taking that next step because they're thinking that they're […] not strong in that area” (SC4).

CFIR Construct: Reflecting and Evaluating

Protected Space for Provider Reflection and Learning

Most SC assigned to the intervention group perceived their monthly meetings as very helpful (33%) or helpful (67%). Some PL (n = 2) and one SC elaborated on the importance of creating an environment where team members “[come] together, virtually or in some sort of chat” (PL1) to share their concerns, generate ideas, and exchange suggestions. “That's when people realize oh, you're experiencing the same thing” (PL1). However, SC recognized the risk to discussing specific nuances of “some cases […] and other SC [not having] that kid on their caseload” (PL3) and how that might be less relevant to all group members.

Instructional Video for Family Reflection

Most families (89%) reported watching the optional YC-PEM e-PRO instructional video prior to completion. While most families found this implementation strategy to be very helpful (35%) or helpful (47%), some families rated it as somewhat helpful (12%) or not helpful (6%). Most families (n = 4) and some SC (n = 2) explained these findings by describing its benefits including (1) an overview of what to expect, (2) a resource to refer to as needed, (3) encouraging respondents to take their time, (4) supports self-reflection, and (5) supports families that need visual support and help to focus when surrounded by distraction(s).

They explained that the video might have been less helpful to those who “didn't really find the instructions complicated” (F3), those who lacked time given the amount of information families receive before an IFSP, and/or those who are in an inconvenient location to watch it. Most SC (n = 5) and one family emphasized the value of giving families options to counter such issues, “besides the video, something that I can follow through at my own pace” (SC2).

CFIR Constructs: Planning and Executing

Timing and Automating of Processes

All three stakeholders (families [n = 2], SC [n = 8], PL [n = 6]) elaborated on the importance of creating a consistent and compatible timeline for implementing the YC-PEM e-PRO in their workflow. SC (n = 7) and families (n = 2) preferred the YC-PEM e-PRO report being sent to the SC well ahead of the scheduled IFSP meeting, “like maybe a week” (SC1), to ensure its integration in their meeting preparation. However, some stakeholders (families [n = 1], SC [n = 2]) recognized the reduced value when YC-PEM e-PRO results are generated too far in advance of the meeting.

Most SC (n = 6) and some PL (n = 2) described instituting flexible timelines, such as asking for the report within “about 24 hours or 48 hours” (SC9) of the IFSP meeting and then allowing those families a second option to “login like 10 or 15 minutes early for the annual” (SC5) to complete the YC-PEM e-PRO during this first part of their IFSP meeting time.

Despite these differences, all three stakeholders (families [n = 2], SC [n = 8] and PL [n = 2]) spoke to the value of having these results “fresh in your mind and having it right when you go into the meeting” (SC6) to ensure their meaningful integration into the IFSP meeting. Some caregivers (n = 2) described wanting more information and instruction to complete the YC-PEM e-PRO ahead of the IFSP meeting. SC (n = 3) and PL (n = 5) suggested further automating processes in ways that promote YC-PEM e-PRO use (e.g., send automated meeting reminders and prompts to complete the YC-PEM e-PRO ahead of the IFSP meeting and “resent to them [the copy of the report] the day before [the meeting] or something” [SC3]).

Optional or Required

Most SC (n = 5) believed optional completion would be more family-centered but recognized the potential in providing choice such as in the timing of completing the YC-PEM e-PRO (e.g., “optional filling it out early but mandatory filling it out”; “they could complete it as part of the IFSP” [SC6]), particularly for families who they anticipate may not opt to complete it but are those that might benefit most from doing so.

CFIR Domain: Inner Setting

For this domain, stakeholders discussed one theme related to the constructs of “culture, network, and communication” and two themes related to the constructs of “structural characteristics and implementation climate” ([Fig. 1]).

CFIR Constructs: Culture, Network, and Communication

Education and Practice for Clear Communication

Most families (n = 5) and SC (n = 6), and a PL (n = 1) described the value of YC-PEM e-PRO results being understood so that interactions and decisions made during IFSP meetings could be justified. Most SC (n = 5) recognized the need for increased practice with the tool, particularly when “coming in as a new staff, you have no idea about it [implementing the YC-PEM e-PRO into your workflow]” (SC3).

CFIR Constructs: Structural Characteristics and Implementation Climate

Incentivizing Organizational Change

A climate that fosters tangible and intangible incentives to its staff (e.g., awards, recognition via performance reviews, salary increases, increased respect) and families (e.g., gift cards) were described by families (n = 1) and SC (n = 4) as creating a strong implementation climate.

Increasing and Changing Workload Demands

Most SC (n = 4) further described their anticipation of tensions that may arise by implementing the YC-PEM e-PRO into their existing workflow, considering available resources (e.g., “space or time for this to happen” [SC4]).

Two PL echoed the difficulties accompanying workflow changes in a program with high productivity demands (e.g., “Even if things are going well, there's just lots of demands” [PL1] and “[losing] a huge number of our SC for turnover” [PL1]) yet recognized the importance of taking part in projects to demonstrate their program's value of research to improve practice.

CFIR Domain: Outer Setting

For this domain, EI stakeholders described themes for the following original and combined constructs: (1) two themes for “cosmopolitanism,” (2) one theme for “patient needs and resources,” and (3) one theme for “external policies and incentives” ([Fig. 1]).

CFIR Construct: Cosmopolitanism

Improving Family-Centered Pathways into EI

PL (n = 3) reflected on the value of YC-PEM e-PRO data for communicating to EI referral sources in ways that improve pathways into EI. They described the use of navigators using data to explain to families the purpose of EI.

Improving Family-Centered Pathway within and across EI Programs Statewide

Most PL (n = 4) elaborated on how the YC-PEM e-PRO illustrates compatibility with state-level measures and initiatives for ensuring quality IFSP development, and strategies for expanding its adoption that they “theoretically […] could do […] with the ARs [annual reviews] right now” (PL2).

CFIR Construct: Patient Needs and Resources

Meeting Needs of Families Accessing EI

Two stakeholders (i.e., SC [n = 6] and PL [n = 5]) explained results depicting trial enrollment for example, mostly White (73%) and non-Hispanic (78%) families with diverse education levels (e.g., 29% earned less than a college degree). PL explained that these demographics aligned with a “pretty large shift in the demographic of Denver as a city” (PL4), which was perceived to affect EI access and use and, in turn, the types of families experiencing the YC-PEM e-PRO. They suggested use of a Spanish version to improve reach.

CFIR Construct: External Policies and Incentives

COVID Restrictions Impacting YC-PEM e-PRO Completion and Family Enrollment

While families (n = 4) expressed difficulty rating their children's participation in community activities when completing the YC-PEM e-PRO during COVID, because of their limited access to those community activities, PL (n = 3) noted increased skills with technology use during COVID.

SC (n = 2) and PL (n = 4) described how COVID and the transition to telehealth impacted family engagement in EI, and consequently, their exposure to the YC-PEM e-PRO and time needed for meaningful engagement due to social isolation.

Discussion

Family engagement is pivotal to the development and implementation of quality e-PROs that can facilitate family-centered service plan development and service improvement. This study elicited perspectives of three key EI stakeholder groups (i.e., families of young children with developmental needs who are enrolled in EI services, EI SC, and EI PL) to fill clinically important knowledge gaps about the full range of considerations for implementing the YC-PEM e-PRO, when paired with a program-specific decision-support tool, to support family-centered service plan development and service improvement. Nineteen themes were identified across all four CFIR domains, revealing relevant supports, barriers, and strategies to implementing this option across EI program workflows.

Intervention and Process Considerations for YC-PEM e-PRO Implementation

Most themes addressed “intervention characteristics” (n = 6) and “processes” (n = 6). For longer-term implementation, EI stakeholders prioritized: (1) its design (e.g., needing resources to tailor and support easy completion, and shaping its potential for data-driven programmatic decision-making); and (2) expanding family and provider engagement (e.g., Latinx families and those with undocumented status and who may not identify as English speaking).

The need and benefits of easy and tailored YC-PEM e-PRO administration as described by stakeholders are critical to informatics tools and consistent with prior work.[17] [18] For example, caregivers have appreciated the comprehensiveness of the YC-PEM e-PRO while recognizing the risk to feeling overwhelmed.[17] Stakeholders specified new ways to tailor YC-PEM e-PRO implementation, by explicitly making the option for completing it and timing its administration to obtain accurate information that can be integrated into service plans. While existing implementation strategies simplified instruction content (e.g., introductory video for families, semi-structured case-based monthly research group meetings) to enable stakeholders to focus and reflect on how they complete or integrate YC-PEM e-PRO results into the IFSP meeting, EI stakeholder groups introduced new ideas about automating processes (e.g., increasing automated reminders for completion, automatically resending a report copy the day before the IFSP meeting). Their ideas yield initial core requirements for the design and development of YC-PEM e-PRO interface(s) with customizable dashboards tailored to the distinct informational needs of each EI stakeholder group.[40] [52] For example, they proposed custom content displays (item-level vs. aggregate YC-PEM e-PRO results) and tailored timing of reminders and report information. Human factors and health informatics expertise may ensure that these interfaces include human-centered data entry and easy-to-comprehend visualizations through co-design and iterative usability evaluation and refinement.[53] [54] [55] [56] [57] Interventions that are tailored appropriately can better fit into workflow and more effectively support user needs and activities, to be responsive to health literacy and cultural needs that can shape patient portal use.[58]

The need to continue expanding engagement to less socially privileged families is also consistent with prior work.[59] To ensure equitable use of the YC-PEM e-PRO, developments are underway to: (1) facilitate cultural adaptations of the tool and its related products (e.g., introductory video), with and without language translation and per best practice guidelines[59]; (2) improve engagement of racially minoritized families to upgrade select content (e.g., centering the expertise of Black and African-American identifying families to introduce anti-racist terms within the YC-PEM e-PRO); and (3) extend its functionality by pairing it with the PEM + . PEM+ is a goal-setting application that is being automated and personalized to enhance user navigation support online for goal-setting purposes. It has similar advantages to the YC-PEM e-PRO for families,[31] [32] [60] such as the relative advantage for capturing family priorities ahead of an IFSP meeting, and can be expected to guide a more efficient and meaningful meeting experience.

Inner and Outer Setting Considerations for YC-PEM e-PRO Implementation

There were relatively fewer themes specific to “inner setting” (n = 3) and “outer setting” (n = 4). For longer-term implementation, stakeholders prioritized: (1) culture, network and communication (e.g., education and practice for clear communication), and (2) cosmopolitanism (e.g., improving family-centered pathways into EI and within and across EI programs statewide). Taken together, stakeholders were prompted to consider ways to better communicate with families receiving EI services and those who might benefit from these services. This finding may represent opportunities to broaden the value of the YC-PEM e-PRO for prospective and current families across EI programs. Prior evidence shows that the YC-PEM e-PRO option is viable for use across EI programs with varying levels of research culture, but this evidence is restricted to its use with current families in a single service catchment.[27] [61] Since EI programs vary considerably in their workflows, available resources, and readiness for change, implementing participation-focused innovations like the YC-PEM e-PRO may benefit from formal assessment of organizational culture and readiness for change, availability of resources for research partnership, and caseload and time constraints within the organization and broader service catchment.[33] [34] We therefore can and should anticipate ample scope for further work to design hybrid type-2 and -3 approaches for testing the implementation of a YC-PEM e-PRO option within a broader EI service system, guided by systematic baseline assessment of each participating EI program and according to: (1) its research culture (e.g., Community Impacts of Research Oriented Partnerships [CIROP][62]; and (2) its practice behaviors such as adoption of participation-focused practices that are reinforced by tools like the YC-PEM e-PRO option (e.g., Method for using Audit and feedback in Participation implementation[63] [64]; Professional Evaluation and Reflection on Change Tool[65]). Such programmatic assessment could help with testing how well the YC-PEM e-PRO performs in different conditions, how to leverage available resources to incentivize and prepare an EI site for implementation, and how to monitor service outcomes.

Work is also underway to examine mechanisms of family-centered service coordination in broader EI service catchments across multiple states, to further elucidate the range of EI workflows to be assessed in future trials.[3] In the larger trial and related work to examine family-centered EI care mechanisms, we intentionally focus on strengthening strategies for recruiting, retaining, and describing underrepresented families in EI who enroll in our research, to support a more inclusive EI research experience.[66] We anticipate its increasing value in future research phases, due to mixed evidence about the role of health literacy in patient portal use.[58] [67]

Limitations

There are study limitations related to the timing of data collection during the COVID 2019 pandemic. First, this pandemic prolonged pragmatic trial data collection, resulting in the need to present interim quantitative data on implementation to drive qualitative data collection with the resources available for this project. While final distribution of quantitative estimates does not significantly differ from the distribution of interim estimates presented to stakeholders, using interim results during data collection could have underestimated the magnitude of findings that stakeholders were prompted to explain in the qualitative phase. Second, we included a small number of participants and a lower percentage of families relative to prior research,[18] thereby potentially limiting saturation of some thematic findings. Families may have experienced increased response burden while being recruited during their trial enrollment lasting up to 12 months. Despite this limitation, most themes reached saturation.

Conclusion

This mixed-methods study focused on implementation outcomes by reinforcing and expanding EI stakeholder priorities for supports, barriers, and strategies to implementing the YC-PEM e-PRO option into diverse EI workflows. Critical to implementation of health informatics tools is the engagement of multiple stakeholder perspectives. Stakeholders positively appraised existing implementation strategies specific to the intervention, their process of delivering it, and their inner organizational culture relative to the realities of their outer setting. They also identified three new priorities for improving YC-PEM e-PRO implementation: (1) upgrade the intervention by developing versions and functionality to expand its advantage for a broader range of families with diverse linguistic preferences and user navigation needs[56]; (2) upgrade the process by further tailoring interface(s) to obtain and use reported information to inform service design and programmatic decisions[52]; and (3) evaluate strategies for building a positive implementation climate (within the organization and externally) to champion and contribute to data-driven changes. These three stakeholder priorities will guide intervention, process, and interface development prior to undertaking pragmatic trials to test the effectiveness of YC-PEM e-PRO implementation across multiple EI programs that share electronic data capture capabilities but differ in their organizational readiness and access to internal and external resources for implementation.

Clinical Relevance Statement

The YC-PEM e-PRO is an evidence-based and promising electronic patient-reported outcome tool for promoting family-centered and participation-focused EI services. Stakeholder engaged research involving EI enrolled families, providers, and program leadership is important for thorough evaluation of its viability for implementation to foster collaborative relationships between EI service providers and families, and to guide data-driven programmatic decision-making by EI program leadership. Results of this mixed-methods study illustrate its usability and scalability, when paired with a decision support tool, both within and across local and state EI programs with electronic data capture systems. Results also highlight meaningful and focused opportunities to engage human factors and health informatics expertise in subsequent phases of research to test the implementation of the YC-PEM e-PRO as an informatics tool in EI. This expertise can be used to create interface(s) with customizable dashboards for personalizing how information from families is obtained and used to guide decisions about service plan development and service improvement. These dashboards might include customized content displays, tailored timing of reminders and report information, and human-centered data entry and easy-to-comprehend visualizations.

Human Subject Protection

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by the Institutional Review Board at the University of Illinois (protocol #2020–0555) and the University of Colorado (protocol #20–2380).

Multiple-Choice Questions

1. Which of the following Inner and Outer Setting constructs were most prioritized by all stakeholder groups for longer-term implementation of the YC-PEM e-PRO:

   • Inner setting: implementation climate; Outer setting: patient needs and resources.

   • Inner setting: culture, network and communication; Outer setting: cosmopolitanism.

   • Inner setting: external policies and incentives; Outer setting: structural characteristics.

   • Inner setting: reflecting and evaluating; Outer setting: relative advantage.

   Correct Answer: The correct answer is option b.

   For Inner Setting, culture, network and communication (e.g., education and practice for clear communication) was most prioritized by stakeholders when considering longer-term implementation. Stakeholders highlighted both, the need for clear communication within the EI program as well as across EI programs. For Outer Setting, cosmopolitanism (e.g., improving family-centered pathways into EI and within and across EI programs statewide) was most prioritized by stakeholders when considering longer-term implementation. Taken together, stakeholders were prompted to consider ways to better communicate with prospective and current families receiving EI services. The use of digital tools like YC-PEM e-PRO for improving family-centered pathways into and through EI, requires sensitivity to broader systemic conditions, particularly for multiply marginalized families.

2. Which of the following Intervention Characteristics and Process constructs were most represented by all stakeholder groups for longer-term implementation of the YC-PEM e-PRO:

   • Intervention characteristics: intervention source; Process: adaptability and trialability.

   • Intervention characteristics: evidence strength and quality; Process: peer pressure.

   • Intervention characteristics: complexity, design and packaging; Process: engaging.

   • Intervention characteristics: reflecting and evaluating; Process: relative advantage.

   Correct Answer: the correct answer is option c.

   For intervention characteristics, EI stakeholders prioritized complexity, design, and packaging (i.e., resources to tailor administration and access to results, use of skip logic and automated processes to make it easier to complete) and providing opportunities for data to drive individual client and programmatic decisions. For Process, stakeholders also prioritized expanding family (e.g., Latinx, undocumented families or families who do not speak English as their primary language) and provider engagement (e.g., organizational culture, readiness for change, resources for research partnerships, and caseload and time constraints).

Conflict of Interest

The YC-PEM e-PRO was used in this study and is licensed for distribution through CanChild Centre for Childhood Disability Research. Dr. Mary Khetani shares in revenue from YC-PEM sales for research and development activities in her laboratory.

Protection of Human and Animal Subjects

This manuscript does not include any research on human subjects.

^† These authors have contributed equally to this work and share first authorship.

^†† These authors have contributed equally to this work and share senior authorship.

• References

• 1 Individuals with Disabilities Education Act, Reauthorization. Public Law 108–446. 2004: 99–457
• 2 US Department of Education. 2020 Part C SPP/ APR indicator analysis booklet. Accessed June 30, 2022, at: https://sites.ed.gov/idea/files/PartC-IndicatorAnalysis-FFY2018.pdf
• 3 Magnusson DM, Khetani MA. Early intervention. In: Blum NJ, Jimenez M. eds. Developmental-Behavioral Pediatrics. 5th ed.. Elsevier;
• 4 Magnusson DM, Mistry KB. Racial and ethnic disparities in unmet need for pediatric therapy services: the role of family-centered care. Acad Pediatr 2017; 17 (01) 27-33
• 5 McManus BM, Richardson Z, Schenkman M. et al. Child characteristics and early intervention referral and receipt of services: a retrospective cohort study. BMC Pediatr 2020; 20 (01) 84
• 6 Richardson ZS, Khetani MA, Scully E, Dooling-Litfin J, Murphy NJ, McManus BM. Social and functional characteristics of receipt and service use intensity of core early intervention services. Acad Pediatr 2019; 19 (07) 722-732
• 7 Jimenez ME, Fiks AG, Shah LR. et al. Factors associated with early intervention referral and evaluation: a mixed methods analysis. Acad Pediatr 2014; 14 (03) 315-323
• 8 Boop C, Cahill SM, Davis C. et al. Occupational therapy practice framework: domain and process 4th ed. Am J Occup Ther 2020; 74 (02) 7412410010p1-7412410010p87
• 9 Ronen GM, Rosenbaum PL, Streiner DL. Patient perspectives in pediatric neurology: a critical shift in the paradigm of outcome measurement. Dev Med Child Neurol 2022; 64 (02) 149-155
• 10 Stern BZ. Clinical potential of patient-reported outcome measures in occupational therapy. Am J Occup Ther 2022; 76 (02) 1-5
• 11 Keeney T, Kumar A, Erler KS, Karmarkar AM. Making the case for patient-reported outcome measures in big-data rehabilitation research: implications for optimizing patient-centered care. Arch Phys Med Rehabil 2022; 103 (5S): S140-S145
• 12 Austin E, LeRouge C, Hartzler AL, Chung AE, Segal C, Lavallee DC. Opportunities and challenges to advance the use of electronic patient-reported outcomes in clinical care: a report from AMIA workshop proceedings. JAMIA Open 2019; 2 (04) 407-410
• 13 Jarvis JM, Fink EL. More than a feeling: understanding function and health related quality of life after pediatric neurocritical illness. Neurocrit Care 2021; 35 (02) 308-310
• 14 Hsiao CJ, Dymek C, Kim B, Russell B. Advancing the use of patient-reported outcomes in practice: understanding challenges, opportunities, and the potential of health information technology. Qual Life Res 2019; 28 (06) 1575-1583
• 15 Kwok EYL, Rosenbaum P, Thomas-Stonell N, Cunningham BJ. Strengths and challenges of the COSMIN tools in outcome measures appraisal: a case example for speech-language therapy. Int J Lang Commun Disord 2021; 56 (02) 313-329
• 16 Khetani MA, Coster W, Law MC, Bedell GM. Young Children's Participation and Environment Measure. Colorado State University (copyright to authors); 2013
• 17 Khetani MA, McManus BM, Arestad K. et al. Technology-based functional assessment in early childhood intervention: a pilot study. Pilot Feasibility Stud 2018; 4 (01) 65
• 18 Albrecht EC, Kaelin VC, Rigau BL. et al; High Value Early Intervention Research Group. Pilot implementation of an electronic patient-reported outcome measure for planning and monitoring participation-focused care in early intervention. BMC Med Inform Decis Mak 2020; 20 (01) 199
• 19 Khetani MA, Graham JE, Davies PL, Law MC, Simeonsson RJ. Psychometric properties of the young children's participation and environment measure. Arch Phys Med Rehabil 2015; 96 (02) 307-316
• 20 Lim CY, Law M, Khetani M, Rosenbaum P, Pollock N. Psychometric evaluation of the young children's participation and environment measure (YC-PEM) for use in Singapore. Phys Occup Ther Pediatr 2018; 38 (03) 316-328
• 21 Khetani MA. Validation of environmental content in the Young Children's Participation and Environment Measure. Arch Phys Med Rehabil 2015; 96 (02) 317-322
• 22 Benjamin TE, Lucas-Thompson RG, Little LM, Davies PL, Khetani MA. Participation in early childhood educational environments for young children with and without developmental disabilities and delays: a mixed methods study. Phys Occup Ther Pediatr 2017; 37 (01) 87-107
• 23 Khetani MA, Cohn ES, Orsmond GI, Law MC, Coster WJ. Parent perspectives of participation in home and community activities when receiving Part C early intervention services. Top Early Child Spec Educ 2013; 32 (04) 234-245
• 24 Khetani M, Marley J, Baker M. et al. Validity of the participation and environment measure for children and youth (PEM-CY) for health impact assessment (HIA) in sustainable development projects. Disabil Health J 2014; 7 (02) 226-235
• 25 Gurga A, Greif A, Sidorowych C, Magasi S, Khetani MA. Success Story: Designing a Participation-Focused Report Yield. Accessed July 1, 2022, at: https://www.canchild.ca/en/resources/223-young-children-s-participation-and-environment-measure-ycpem
• 26 Harrison JD, Auerbach AD, Anderson W. et al. Patient stakeholder engagement in research: a narrative review to describe foundational principles and best practice activities. Health Expect 2019; 22 (03) 307-316
• 27 Martinez J, Wong C, Piersol CV, Bieber DC, Perry BL, Leland NE. Stakeholder engagement in research: a scoping review of current evaluation methods. J Comp Eff Res 2019; 8 (15) 1327-1341
• 28 Chaudoir SR, Dugan AG, Barr CHI. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci 2013; 8 (01) 22
• 29 Khetani MA, Cliff AB, Schelly C, Daunhauer L, Anaby D. Decisional support algorithm for collaborative care planning using the Participation and Environment Measure for Children and Youth (PEM-CY): a mixed methods study. Phys Occup Ther Pediatr 2014; 35 (03) 231-252
• 30 Khetani MA, Lim HK, Corden ME. Caregiver input to optimize the design of a pediatric care planning guide for rehabilitation: descriptive study. JMIR Rehabil Assist Technol 2017; 4 (02) e10
• 31 Jarvis JM, Gurga AR, Lim H. et al. Caregiver strategy use to promote children's home participation after pediatric critical illness. Arch Phys Med Rehabil 2019; 100 (11) 2144-2150
• 32 Jarvis JM, Kaelin VC, Anaby D, Teplicky R, Khetani MA. Electronic participation-focused care planning support for families: a pilot study. Dev Med Child Neurol 2020; 62 (08) 954-961
• 33 Rigau BL, Scully EA, Dooling-Litfin JK, Murphy NJ, McManus BM, Khetani MA. Community engagement to pilot electronic patient-reported outcomes (e-PROs) in early intervention: lessons learned. J Clin Transl Sci 2018; 2 (01) 20-26
• 34 Anaby D, Khetani M, Piskur B. et al. Towards a paradigm shift in pediatric rehabilitation: accelerating the uptake of evidence on participation into routine clinical practice. Disabil Rehabil 2022; 44 (09) 1746-1757
• 35 Elbaum B, Celimli-Aksoy S. Developmental outcomes of children served in a Part C early intervention program. Infants Young Child 2022; 35 (01) 3-19
• 36 Khetani MA, Bosak D, Jarvis JM, Teplicky R. Young Children' s Participation and Environment User Guide (Version 1.2). Accessed July 1, 2022 at: https://www.canchild.ca/en/shop/23-yc-pem-young-children-s-participation-and-environment-measure
• 37 McManus BM, Murphy NJ, Richardson Z, Khetani MA, Schenkman M, Morrato EH. Family-centred care in early intervention: examining caregiver perceptions of family-centred care and early intervention service use intensity. Child Care Health Dev 2020; 46 (01) 1-8
• 38 Papautsky EL, Patterson ES. Patients are knowledge workers in the clinical information space. Appl Clin Inform 2021; 12 (01) 133-140
• 39 Papautsky EL, Crandall B, Grome A, Greenberg JM. A case study of source triangulation. J Cogn Eng Decis Mak 2015; 9 (04) 347-358
• 40 Kaelin V, Villegas V, Chen YF. et al; High Value Early Intervention Research Group. Effectiveness and scalability of an electronic patient-reported outcome measure and decision support tool for family-centred and participation-focused early intervention: PROSPECT hybrid type 1 trial protocol. BMJ Open 2022; 12 (01) e051582
• 41 Creswell J, Clark V. Designing and Conducting Mixed Methods Research. 2nd ed.. Sage Publications, Inc.; 2011
• 42 Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012; 50 (03) 217-226
• 43 Landes SJ, McBain SA, Curran GM. Reprint of: an introduction to effectiveness-implementation hybrid designs. Psychiatry Res 2020; 283: 112630
• 44 Lambert SD, Loiselle CG. Combining individual interviews and focus groups to enhance data richness. J Adv Nurs 2008; 62 (02) 228-237
• 45 QSR International Pty Ltd NVivo. . Accessed July 1, 2022, at: https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home
• 46 Mayring P. Qualitative Content Analysis: Theoretical Foundation, Basic Procedures and Software Solution. Klagenfurt; 2014
• 47 Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4 (01) 50
• 48 Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci 2015; 10 (01) 53
• 49 Krueger RA, Casey MA. Focus Groups: A Practical Guide for Applied Research. 5th ed.. Sage Publications, Inc.; 2015
• 50 US Department of Housing and Urban Development. Estimated Median Family Incomes for Fiscal Year (FY) 2021. Accessed July 1, 2022 at: https://www.huduser.gov/portal/datasets/il/il21/Medians2021.pdf
• 51 Khetani MA, Kaelin VC, Kuznicki L, Bane J, Banks S. Academic-community partnership to improve early intervention. Collaborative Endeavors. Accessed July 1, 2022 at: https://collaborative-endeavors.simplecast.com/episodes/khetani-txJ9ZBT0
• 52 Galvin HK, Petersen C, Subbian V, Solomonides A. Patients as agents in behavioral health research and service provision: recommendations to support the learning health system. Appl Clin Inform 2019; 10 (05) 841-848
• 53 Russ AL, Fairbanks RJ, Karsh BT, Militello LG, Saleem JJ, Wears RL. The science of human factors: separating fact from fiction. BMJ Qual Saf 2013; 22 (10) 802-808
• 54 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
• 55 Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academies Press: 2001
• 56 Swoboda CM, DePuccio MJ, Fareed N, McAlearney AS, Walker DM. Patient portals: useful for whom and for what? A cross-sectional analysis of national survey data. Appl Clin Inform 2021; 12 (03) 573-581
• 57 van Leeuwen D, Mittelman M, Fabian L, Lomotan EA. Nothing for me or about me, without me: codesign of clinical decision support. Appl Clin Inform 2022; 13 (03) 641-646
• 58 Di Tosto G, Walker DM, Sieck CJ. et al. Examining the relationship between health literacy, health numeracy, and patient portal use. Appl Clin Inform 2022; 13 (03) 692-699
• 59 Tomas V, Srinivasan R, Kulkarni V, Teplicky R, Anaby D, Khetani M. A guiding process to culturally adapt assessments for participation-focused pediatric practice: the case of the Participation and Environment Measures (PEM). Disabil Rehabil 2022; 44 (21) 6497-6509
• 60 Lyman J. The participation and environment measure-plus: a useful new tool for caregivers and professionals. Dev Med Child Neurol 2020; 62 (08) 890-890
• 61 Rigau BL, Dooling-Litfin J, Scully E. et al. Building organizational capacity for research in early intervention. Zero Three 2019; 39 (06) 54-61
• 62 King G, Servais M, Kertoy M. et al. A measure of community members' perceptions of the impacts of research partnerships in health and social services. Eval Program Plann 2009; 32 (03) 289-299
• 63 Graham F, Timothy E, Williman J, Levack W. Participation-focused practices in paediatric rehabilitation for children with neurodisability in New Zealand: an observational study using MAPi audit tool. Child Care Health Dev 2020; 46 (05) 552-562
• 64 Kolehmainen N, Marshall J, Hislop J. et al. Implementing participation-focused services: a study to develop the method for using audit and feedback in participation implementation (MAPi). Child Care Health Dev 2020; 46 (01) 37-45
• 65 Menon A, Cafaro T, Loncaric D. et al. Creation and validation of the PERFECT: a critical incident tool for evaluating change in the practices of health professionals. J Eval Clin Pract 2010; 16 (06) 1170-1175
• 66 Charles and Lynn Schusterman Family Foundation. More than numbers. A guide toward diversity, equity, and inclusion (DEI) in data collection. Accessed July 1, 2022 at: https://cloudspn.tfaforms.net/266232
• 67 Nguyen OT, Hong YR, Alishahi Tabriz A, Hanna K, Turner K. Prevalence and factors associated with patient-requested corrections to the medical record through use of a patient portal: findings from a national survey. Appl Clin Inform 2022; 13 (01) 242-251

Publication History

Received: 14 July 2022

Accepted: 09 December 2022

Article published online:
01 February 2023