0920-24FI In-Depth Assessment – Evaluability Assessment Recipient

Form Approved

OMB No. 0920-XXXX

Exp. Date xx/xx/20xx

In-Depth Assessment – Evaluability Assessment Recipient Interview Guide – Team-Based Care – The National Cardiovascular Health Program & The Innovative Cardiovascular Health Program

Note: Public reporting burden of this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/ATSDR Reports Clearance Officer; 1600 Clifton Road NE, MS D-74, Atlanta, Georgia 30333; ATTN: PRA (0920-24XXX)

Evaluability Assessment

Recipient Interview Guide

Date of Interview
Interviewer
Notetaker
Organization Name
Organization Type
State
Organization City		Zip Code
Cooperative Agreement	☐ The National CVH Program ☐ The Innovative CVH Program
Strategy	Strategy 2: Implement Team Based Care
Interviewee Name(s)
Interviewee Role(s) or Title(s)

Introduction

Thank you for taking the time to participate in this interview. My name is <insert name> and I am with the Deloitte evaluation team. Our team is working with the CDC Division for Heart Disease and Stroke Prevention to evaluate the <insert Cooperative Agreement>. As part of the CDC-led evaluation, we are conducting evaluability assessment interviews to provide detailed insight into how recipients and their partners are prioritizing populations of focus impacted by the high prevalence of cardiovascular disease through Strategy 2: Implement Team Based Care. We hope to learn about the function, structure, goals, and activities of your program in today’s discussion. Additionally, the evaluability assessment will be used to identify recipients and partners with promising approaches, who will be invited to participate in an exploratory assessment during PY4.

Our team has drafted a logic model based on program materials that your team shared with us prior to this interview. We may refer to the draft logic model throughout the interview to facilitate discussion on program goals, activities, desired outcomes, and contextual factors.

This interview is expected to take no longer than 90 minutes. Please answer questions based on your own knowledge and experience. Remember, you are the expert and that there are no right or wrong answers. If at any time during the interview you are not clear about what we are asking, be sure to let me know. Your participation in this interview is completely voluntary. You may choose not to respond to questions at any time and it will not in any way impact the funding or technical assistance your organizations receive from CDC.

Steps will be taken to protect your privacy; no information that identifies you will be shared with anyone except our project staff. All information will be kept secure and any personally identifiable information will be removed when results are aggregated for analysis.

Do you consent to this interview?

☐ Yes

☐ No

With your permission, we would like to record this interview for transcription purposes. 

Do we have your permission to record?

​​☐​ Yes 

​​☐​ No 

 

Do you have any questions or concerns before we start the discussion? 

Background

Thank you again for participating in this interview. For reference, today’s interview we will be talking about Strategy 2, which is defined as:

[Interviewer Note: Only describe the relevant sub-strategies for which the recipient organization has self-nominated.]

Implement team-based care to prevent and reduce CVD risk with a focus on hypertension and high cholesterol prevention, detection, control, and management through the mitigation of social support barriers to improve outcomes. (The National CVH Program)

Implement team-based care to prevent, detect, control, and manage hypertension and high cholesterol within approved populations of focus. (The Innovative CVH Program)

We will discuss the following sub-strategies under Strategy 2:

2A: Advance the use of health information systems that support team-based care to monitor population health with a focus on health disparities, hypertension, and high cholesterol [within approved populations of focus. (The Innovative CVH Program)]

2B: Assemble or create multidisciplinary teams (e.g., nurses, nurse practitioners, pharmacists, nutritionists, physical therapists, social workers, and community-based workers) to identify patients' social services and support needs.

• and to improve the management and treatment of hypertension and high cholesterol (The National CVH Program)

• within approved populations of focus (The Innovative CVH Program)

2C: Build and manage a coordinated network of multidisciplinary partnerships that address identified barriers.

• to social services and support needs (e.g., childcare, transportation, language translation, food assistance, and housing) within populations at highest risk of CVD (The National CVH Program)

  • and needs within approved populations of focus, related to their social services and support needs (e.g., childcare, transportation, language translation, food assistance, and housing) (The Innovative CVH Program)

First, we would like to learn a little about you and your organization.

[Interviewer Note: Use the following question to confirm information learned from the nomination form and document review about interviewee’s organization.]

1. From the <nomination form, APR, work plan, EPMP, etc.>, we learned that your organization offers <programs and services> for <population >. Is this correct? Is there anything else you would like to add or clarify?

Probes:

• What types of services/programs, related to implementing team-based care, does your organization offer to support individuals who have or at high risk for high blood pressure or high cholesterol?

• How long has your organization been <offering these services, implementing these programs, providing this support>?

• What are the overall goals of these programs? What is your organization hoping to achieve through these programs?

• Can you describe to me the different populations (i.e., race, ethnicity, socioeconomic status, age, genders, geography, <census tracts for Innovative CVH Program>) that your organization typically serves or focuses on?

[Interviewer Note: Use the following question to understand the interviewee’s role related to the nominated strategy/sub-strategies.]

2. What is/are your role(s) and what are your specific responsibilities related to <name of NOFO>?

Probes:

• How long have you been working with <organization name>?

• How long have you been in this role?

• Can you tell me about your role in relation to supporting the implementation of team-based care?

Program Implementation

[Interviewer Note: Ask about each nominated sub-strategy for the Program Implementation questions.]

Next, we would like to discuss your organization’s approach to address team-based care or TBC. We’re interested in learning more about the program goals, key activities, implementation strategy, and intended program reach.

[Interviewer Note: Use the following question to understand the implementation of the nominated strategy/sub-strategies. Confirm what we’ve learned from the document review and nomination form. Tailor the language based on how recipient refers to their program and activities rather than using NOFO specific language]

3. According to the <organization’s program materials, recipient-led evaluation deliverables, nomination form>, your program approach related to TBC is <description of program. > Is this correct? Can you tell us more about the key activities and core components of <program name>? Describe things like the types of interventions being implemented, how it is implemented, and in what settings.

Probes:

• Where is the intervention implemented?

• 2A: How does your program approach increase the use of EHRs/HIT to support TBC?

  • How will data be used for TBC?

  • What are the steps involved?

• 2B: What does TBC look like at your partner organizations?

  • How do care team members work with patients?

• 2C: How do coordinated networks of multidisciplinary partners work to address the needs of patients?

• 2C: When building and managing networks of multidisciplinary partnerships, how did you identify organizations that address barriers to social services and support needs?

  • What types of partners are part of these networks?

  • What are the gaps in these networks, if any?

[Interviewer Note: Ask the following questions if the recipient stated in the nomination form that they are working on cardiac rehab, or hypertension among women, or hypertension among pregnant or postpartum people.]

4. According to the < nomination form>, we learned that you are implementing <cardiac rehab and/or activities related to hypertension in women and/or activities related to hypertension in pregnant or postpartum people.> Can you tell us more about these activities?

Probes:

• [If applicable based on recipient response in the nomination form] Tell me more about your cardiac rehab programming. What types of activities are implemented?

• [If applicable based on recipient response in the nomination form] What types of intervention activities prioritize or focus on pregnant or postpartum people? How do you tailor your activities for pregnancy and postpartum period?

• [If applicable based on recipient response in the nomination form] What types of intervention activities prioritize or focus on women? How do you tailor your activities for women?

5. What partnerships are in place to support <TBC strategy/sub-strategy implementation>?

Probes:

• What types of organizations are you partnering with and in what ways do they implement the program?

• Tell me more about how you collaborate with your partners.

• How do you support your partners to implement their TBC activities?

• What has worked well and what hasn’t worked as well with your partnerships?

1. From your <program review documents and nomination form>, we understand that your learning collaborative <description of LC activities >. Is that accurate? Tell us more about the role of the learning collaborative (LC) in <TBC activities>?

Probes:

• How does the LC support strategy implementation?

• How does the LC influence partnership networking? What about program reach?

7. What are the goals of the <program name related to strategy> and how will the <program> achieve these goals?

Probe:

• What do you hope to achieve through TBC activities?

8. What is the <program’s> population of focus? What demographics, patient characteristics, or geographies are you prioritizing?

Probes:

• How do you define populations at highest risk of CVD? (The National CVH Program)

• How did you identify your population of focus? What data and methods did you use?

• What tools and resources have you used to understand or identify the health disparities in your population of focus?

• What barriers do the population(s) of focus face in terms of management and treatment of CVD? How do SDOH factors affect their CVD risk?

9. How does your <program> focus on <name of population of focus>? Please describe how program activities are tailored to the needs of your population.

Probes:

• How do SDOH affect patient engagement with TBC? How does this inform your program approach?

• What challenges has your program experienced with engaging patients in TBC?

• What strategies have worked best for engaging the population(s) of focus in TBC?

• What innovations are implemented to address SDOH? (The Innovative CVH Program)

Now that we’ve learned more about your program approach from its goals and operations, we would like to learn more about the extent of program implementation, successes and challenges with implementation, and factors that may support or hinder activities.

10. Tell us more about your progress related to <name of program> and <TBC activities>?

Probe:

• Tell us more about milestones and other achievements.

• What are some areas where you did not make as much progress as anticipated?

11. What are your future implementation plans?

Probes:

• How will your partnerships change or grow in the next few years?

• Please describe what you hope to accomplish by Year 4 (September 2026 – September 2027).

12. Can you tell us about the contextual factors that support or hinder activities related to strengthening TBC?

Probes:

• Describe external factors such as complementing or competing initiatives, additional funding sources, partnerships and collaborations, state policies, political/economic climate.

• Describe internal factors such as organizational policies, leadership buy-in, internal capacity, organizational culture.

13. What would you say are the strengths of your <program>?

Probe:

• What factors positively affect <TBC strategy/sub-strategy implementation> or helped the program be successful?

1. What challenges have you experienced with < TBC strategy/sub-strategy implementation>?

Probes:

• How are these challenges addressed?

• What support, resources, or TA do you need to overcome these barriers?

15. What have been some key lessons learned from your experience with <TBC strategy/sub-strategy implementation>?

[Interviewer Note: Only ask the next set of questions if the recipient organization participates in more than one NOFO. Otherwise, move on to the Program Evaluation section.]

Next, we’re interested in learning more about your organization’s involvement in other DHDSP cooperative agreement programs or how your organization may coordinate with other recipient organizations.

16. What other DHDSP cooperative agreements do you receive funding from? For example, <The National CVH Program, The Innovative CVH Program, WISEWOMAN program.>

17. Do you coordinate with other recipient organizations that receive funding from DHDSP cooperative agreements?

18. How are you coordinating across <NOFO name(s)> to maximize resources and avoid duplication of effort?

Probes:

• How does your organization leverage funding across NOFOs?

• How does your organization leverage partnerships across NOFOs?

• How has your organization coordinated resources for shared impact across NOFOs?

• [If implementing both The National and Innovative CVH Programs] How are you coordinating learning collaboratives The National and Innovative CVH Programs?

19. What are the advantages of participating in and/or coordinating multiple NOFOs?

20. What are the challenges with participating in and/or coordinating multiple NOFOs?

Program Evaluation

[Interviewer Note: Ask about each nominated sub-strategy for the Program Evaluation questions.]

We would like to understand to what extent the <TBC activities> have been or are currently being evaluated. We are also interested in learning about your organization’s capacity to evaluate <program/activities>.

1. Who is primarily responsible for tracking and reporting data related to <TBC implementation> for <name of cooperative agreement>?

Probes:

• What is your role in data collection and monitoring and evaluation efforts?

• Are other members or organizations involved in data collection? Describe their roles and responsibilities.

• Is the LC involved in data collection and evaluation efforts? How so?

2. What data are collected to help monitor <program activities> and evaluate its success?

Probes:

• What types of metrics or indicators do you use to measure progress and monitor implementation?

  • [If the interviewee only mentions PMs] What about metrics other than the NOFO performance measures?

• How is success measured?

• What outcomes do you track? What outcomes do you expect to have by September 2026 – September 2027 (Y4)?

  • [If the interviewee only mentions PMs] What about outcomes beyond the NOFO performance measures?

• Does <program> collect data on hypertension control among women? Is your organization willing to provide CDC with hypertension control data by sex?

3. How does <name of recipient> collect the data needed for monitoring and evaluation of TBC?

Probes:

• What data collection tools or instruments are used to track data (paper, electronic)?

  • [If not already provided] Request to see and get copies of data, data collection tools, or evaluation reports.

• Describe the process used to collect these data.

• Describe how <name of recipient> uses these data.

• Do you collect data at specified points over time (time series)? What length of time? How frequently?

4. How is health equity incorporated in your evaluation plan?

Probes:

• What SDOH data do you collect?

• How will health disparities be measured and defined?

• What sort of methods or tools are used to measure health equity outcomes?

5. What, if any, barriers have you encountered with data collection or monitoring and evaluation activities?

Probes:

• What challenges, if any, are there with collecting SDOH data?

• What barriers, if any, do you anticipate in data collection or reporting related to outcome or impact?

• What specific strategies have been used or will you use to overcome these barriers?

• What additional support or TA do you need from CDC?

• What have been some lessons learned from your experience with data collection and evaluation related to TBC for the <name of cooperative agreement>?

6. What have you learned so far about your <TBC activities> from your monitoring and evaluation efforts?

Probes:

• What findings can you share about implementation progress?

• What outcomes can you report at this point?

• How are data being used to make improvements?

7. What is the timeline for the next phase of evaluation?

Probes:

• What are key events for data collection and evaluation?

• Are there anticipated barriers for the next phase?

8. Do you provide evaluation results related to <TBC strategy/sub-strategy> to external audiences such as funders, partners, decision makers, constituents, or others?

Probes:

• What types of findings do you share? How do you share your findings?

• Do they use any of the information you provide? In what sorts of ways?

9. If <name of recipient> is selected to participate in an exploratory assessment of your program, to what extent would <name of recipient> have the capacity to contribute to detailed data collection on outcomes and/or cost?

Probes:

• What kind of assistance do you think you would need?

• Are there any other important considerations we should know about your readiness/capacity to participate in an evaluation?

Closing

Lastly, what questions do you have for me? Is there anything else you’d like to share?

Thank you again for participating. This concludes our discussion about TBC implementations. If you have any additional questions, please feel free to contact the Comprehensive Evaluation Team, hdsp_nofo_eval@cdc.gov.