0920-23AE CDC VDAT HD Customer Satisfaction Survey

Centers for Disease Control and Prevention (CDC)

CDC Vaccine Distribution and Administration Tracking Help Desk 2.0

Customer Satisfaction Survey

Submitted by:

MAXIMUS Federal

3120 Fairview Park Drive

Falls Church, VA 22042

Contract Number: 75D30122F13256

Document Type: Contractual

Document Classification: Confidential and Proprietary

Version 2.0

June 24, 2022

Document Control

DOCUMENT INFORMATION

Title:	CDC VDAT HD Customer Satisfaction Survey
Version:	2.0
Issue Date:	06/24/2022
Security Level:	Confidential and Proprietary
Filename:	CDC VDAT HD Customer Satisfaction Survey_v2.0

DOCUMENT HISTORY

Version	Date	Responsible Individual	Summary of Changes
1.0	02/16/2022	Paul Miranda	Initial Document
2.0	06/24/2022	Beth G. Tanos	Update to Burden Hours Calculation, Email Triage Process Flow/Steps

Customer Satisfaction Survey

Table of Contents

1.0 Acronyms and Terms 4

2.0 Purpose 5

3.0 Process 6

3.1 Email Survey Process 6

3.2 Estimated Burden Calculation 6

4.0 Duplication of CSAT Surveys Across System Applications 8

5.0 Overview of System Architecture and Processes 9

6.0 Survey Questions 12

7.0 Reporting 14

8.0 Data Collection Activities 15

Appendix A: Survey Invite Email Template 17

List of Tables & Figures

Table 1: Program Acronyms and Terms List 4

Table 2: Data Collection Goals 15

Figure 1: CDC VDAT Systems 9

Figure 2: CDC VDAT Workflow 10

Figure 3: CDC VDAT Sample Survey 12

1.0 Acronyms and Terms

The following table lists all the acronyms and terms used in the Centers for Disease Control and Prevention (CDC) Vaccine Distribution and Administration Tracking (VDAT) Help Desk (HD) Customer Satisfaction (CSAT) Survey document. It also provides a definition for the terms used in this plan.

Table 1: Program Acronyms and Terms List

Acronym or Term	Definition
AWS	Amazon Web Services
CARS	COVID-19 Administration Reporting Systems
CDC	Centers for Disease Control and Prevention
COR	Contracting Officer Representative
CRM	Customer Relationship Management
CSAT	Customer Satisfaction
HD	Help Desk
HDS	Help Desk Specialist
HPoP	Health Partner Order Portal
IVR	Interactive Voice Response
OMB	Office of Management and Budget
PII	Personally Identifiable Information
PRA	Paperwork Reduction Act
SLA	Service Level Agreement
SR	Service Request
TM	Technical Monitor
VAMS	Vaccine Administration Management System
VDAT	Vaccine Distribution and Administration Tracking
VF	Vaccine Finder
WFM	Workforce Management

2.0 Purpose

The purpose of this document is to provide an overview of the CSAT Survey Process established by the CDC VDAT HD Team. The CSAT Survey was developed for implementation as part of the CDC VDAT program’s Quality Assurance Plan. The survey will be used to gather qualitative user and stakeholder feedback on CDC VDAT HD service delivery. The results will be used to provide insights into user perceptions, experiences, and expectations of the program. The results will also provide early warning of issues with service and focus attention on areas where communication, training or changes in customer service operations may improve the overall quality and delivery of service of the HD operations.

The CDC process to protect the public from the burden of gathering information within the parameters of the Paperwork Reduction Act (PRA) was taken into consideration when developing this survey. As a result, the survey will be in the form of questions disseminated via the Salesforce Online survey tool only; no blind canvassing or other systems will be utilized. Users calling into the CDC VDAT HD and users contacting the CDC VDAT HD via email will be the only groups targeted for gathering this information.

3.0 Process

The CDC VDAT HD program will monitor the performance of the Help Desk Specialists (HDS), the training they receive, and the information they provide to ensure it meets the needs of the vaccine distribution vendor users and the public at large. The CSAT Survey Process allows the CDC VDAT Team to develop and administer a survey tool via email on the Salesforce system to collect data regarding HDS performance and the HD experience. The use of the Salesforce tool will be leveraged to accomplish the survey Service Level Agreement (SLA) requirement. The CARS HDS creates Service Request (SR) tickets for both the phone and email channels, which ensures that all users can receive a survey. The Salesforce automated system will send a survey invitation based on a ticket’s status. Producing an automated survey takes any bias out of the process by the HDS and ensures that sincere user comments are obtained.

The survey contains a total of five (5) questions which will be used to gather information to determine the following:

1. How useful is the HD

2. How knowledgeable is the HDS

3. How timely is the resolution

4. How satisfied is the user with the HD

5. Overall feedback or comments to improve the HD experience

The survey is designed to be unintrusive and as simple as possible for any responder to understand. Survey questions will not contain Personally Identifiable Information (PII) and the results will only be used to improve the quality of the CDC VDAT HD program.

3.1 Email Survey Process

For tickets created from emails, an automated email invitation with a link to the online survey questions is sent for all tickets moved to a ‘Closed’ status. The online survey is part of the Salesforce tool and allows for maintaining the tickets and the survey data in one system.

A CARS HD user always has the right to refuse responding to the online survey. The process should not be a burden to the users but should provide an effective and positive experience for COVID-19 vaccination users and providers. We have selected users with ‘Closed’ tickets as the optimal engagement audience. This ensures the users responding are able to provide their complete experience with the CARS HD.

3.2 Estimated Burden Calculation

We calculated the estimated burden based on the reporting data captured since the CARS HD launched operations on May 7, 2021. The CARS HD averages 200 closed tickets per day. Based on the responses gathered, a typical email survey response rate is 10%. The expected completion time for the survey is approximately three (3) minutes. Based on this information, the estimated burden is calculated as 260 hours, as shown in below computation.

• 200 Average Daily Tickets x 10% response rate = 20 responses/day

• Total Completion Time per Day: 3 minutes/response x 20 responses/day = 60 minutes/day

• Total Completion Time per Week: 60 minutes/day x 5 days/week = 300 minutes/week or 5 hours/week

• Total Completion Time for the Year: 5 hours/week x 52 weeks = 260 Survey Burden Hours

4.0 Duplication of CSAT Surveys Across System Applications

Limiting Duplication Efforts

When developing the CSAT survey, the CDC VDAT Team took into consideration the CARS HD users. An assessment was performed across current HD applications (i.e., HPoP, VAMS, VF) to assess redundancies of CSAT surveys being administrated to the user base prior to transition of the HD applications to the CARS HD program.

Application Systems for CDC VDAT HD

Below is a high-level overview of the systems which details current CSAT process.

• Health Partner Order Portal (HPoP) users are jurisdictions, states, federal entities, pharmacies, and clinics who order and track their COVID-19 vaccines, diagnostics, and therapeutics. Through our assessment we confirmed via the previous HPoP system owners that CSAT surveys were not administered to their user base. Since the CARS transition, CSAT surveys will be distributed for the first time. Duplication of CSAT questions is therefore not applicable.

• The Vaccine Administration Management System (VAMS) provides access to users to maintain appointments and information on recipient vaccine dosages for a handful of states, prisons, and organizations. Through our assessment we confirmed via the previous VAMS system owners that CSAT surveys were not administered to their user base. Since the CARS transition, CSAT surveys will be distributed for the first time. Duplication of CSAT questions is therefore not applicable.

• Vaccine Finder (VF) focuses on providing information to the public about where vaccines are available, and tracks usage by pharmacies and clinics. Through our assessment we confirmed via the previous VF system owners that CSAT surveys were not administered to their user base. Since the CARS transition, CSAT surveys will be distributed for the first time. Duplication of CSAT questions is therefore not applicable.

CDC-Info

CDC-Info is a national contact center that offers phone and email support to a user base to help find reliable health information on many health care topics. Based on our interactions and the similarities between CDC-Info and CARS HD as it relates to vaccination responses, we obtained copies of the current CDC-Info CSAT. Based upon review and assessment of the current survey, no similarities were observed across the two (2) surveys for questions presented to the user community. The purpose of the CARS HD CSAT is to evaluate a customer’s overall experience with the HD and to evaluate the HDS performance.

5.0 Overview of System Architecture and Processes

CDC VDAT HD System Platforms 

The VDAT Team uses a variety of system platforms to manage operations, assist users/providers and achieve program objectives.  Below is a high-level overview of the systems used. 

 

• Genesys (telephony system) is a suite of pre-integrated components including voice applications, chat, email, Interactive Voice Response (IVR), and Workforce Management (WFM) tools used for HD operations 

• Salesforce (ticketing system) is an application used to capture Customer Relationship Management (CRM) contacts

and call disposition data for calls/emails being handled by HDS

• Amazon Web Services (AWS) is a secure cloud platform/desktop service that ensures the VDAT Team can work remotely and securely 

• Cornerstone is the eLearning platform used for training facilitation and completion 

• CDC Systems are the vendor specific applications used by HDS for

troubleshooting and problem resolution  

 

Figure 1: CDC VDAT Systems

CDC VDAT Workflow

The VDAT Team provides and maintains a comprehensive HD infrastructure that addresses access, technical, and issue resolution for CDC systems supporting the COVID-19 vaccination response.  Below provides a high-level overview of the CDC VDAT Workflow.

Figure 2: CDC VDAT Workflow

Email Triage and Escalation Process

Email Process

1. The CDC CARS HD email inbox (CARS_Helpdesk@cdc.gov) receives a user’s email.

2. Email Triage Team member reviews the email.

3. Email Triage Team member searches for the email address in Salesforce to verify if the user has contacted CARS HD before.

4. If the user has contacted CARS HD before, their contact details will appear as a contact in Salesforce and the Email Triage Team member then copies the contact details into the new ticket. If the user has not made any contact with CARS HD, the Email Triage Team member creates a new contact for the user.

5. The Email Triage Team member inputs the other required information (e.g., Name, Phone Number, VDAT System, etc.) in the ticket and leaves the ticket in the ‘New’ status.

6. HDS selects and works on the ticket in the ‘New’ status and clicks the SR owner’s name and changes it to their name to reflect their ownership of the ticket.

7. HDS works the issue until complete, provides an email response to the user and changes the ticket to ‘Close’.

8. If HDS is unable to resolve the issue, the ticket is escalated to their PC/Tier-2 HDS for review.

9. PC/Tier-2 HDS advises HDS to either close the ticket or escalate to Tier-3 (System Vendor).

10. If ticket is escalated to Tier-3, the HDS selects the ‘Closed by Escalation’ status and sends an email to the user explaining the ticket has been escalated. An escalation email to the Tier-3/System Vendor organization is automatically generated by Salesforce.

Phone Process

1. User calls the CARS 800 number (833-748-1979) and reaches the CARS HD IVR system.

2. User selects the appropriate CDC VDAT system based on the assistance needed.

3. HDS answers the phone and initiates the troubleshooting process.

4. When the phone is answered, Salesforce will check the user’s contact information against the existing contacts in the database. If a matching contact exists, Salesforce will open the contact page. If no contact exists, Salesforce will open the new contact screen.

5. If Salesforce finds a contact, the HDS verifies the user’s information and confirms that the ticket is closed. HDS creates a ‘New’ ticket for the user’s current issue.

6. HDS inputs the other required information (e.g., Category, Issue Details, etc.) in the ticket based on the user’s information.

7. HDS follows steps 7 – 10 of the Email Process (see above).

6.0 Survey Questions

The below questions were chosen to capture data easily and quickly from the CARS HD users.

Figure 3: CDC VDAT Sample Survey

Usefulness

1. Was this the first time you contacted the Help Desk regarding an issue?

Yes No

B) If “No”, please provide additional detail regarding your experience

Comments:

Trained Staff

Was your Help Desk Specialist knowledgeable about the application?

Yes No

Timeliness

Was your inquiry resolved in a timely manner?

Yes No

Satisfaction

How easily did the Help Desk Specialist resolve your issue?

Not Resolved                                          Resolved

0 1 2 3 4 5 6 7 8 9 10

Overall experience

Considering your overall experience with the CDC CARS Help Desk, how likely are you to recommend?

Very Unlikely                                        Very Likely

0 1 2 3 4 5 6 7 8 9 10

7.0 Reporting

Data captured through our Salesforce system for this survey will be tabulated. All responses returned will be broken down by question results to formulate averages and graphs for data analysis. Conceptual aspects of usefulness, staff’s knowledge, timeliness, satisfaction, and experience are analyzed as part of the collection data. This provides the program with more detailed information on the user’s complete experience with the HDS and the CARS HD program as a whole.

8.0 Data Collection Activities

The Data collection goals listed below were determined based on the target audience the survey will reach, the method for collection, and the information gathered.

Table 2: Data Collection Goals

Target Audience(s)	General Data Collection Methods	Description of Data Collection Activities	Outreach and Engagement Methods	Types of Information to be Collected
Goal 1: Identify opportunities for continuous quality improvements of the Help Desk (i.e., people, processes, and tools)
Covid-19 Jurisdictions, States, Federal Entities, Pharmacies, Clinics, Organizations, and Recipients using VF, HPoP, and VAMS	Conduct an online Salesforce survey	The target data collection period will be February 16, 2022 through February 15, 2023 A Salesforce online survey will be administered The survey will be programmed using Salesforce (survey tool) Individuals will receive an email (template displayed in Appendix A) containing an invitation to participate in the survey, which contains a survey link. The invitation will indicate the survey is voluntary, and the individual participant information will be kept confidential. Consent to participate will be assumed when the individual clicks on the provided link to complete the survey The anticipated response rate is 10% of those who receive the survey Survey results will be compiled and included in a weekly metrics report submitted to the CDC Contracting Officer Representative (COR)/Technical Monitor (TM)	Send emails containing request for input and link to online survey for users who contact the CARS HD	Collecting Dichotomous ('Yes/No') answers, free form responses and rated scaled responses
Goal 2: Evaluate user experience based on observations and perceptions of the CARS Help Desk
Covid-19 Jurisdictions, States, Federal Entities, Pharmacies, Clinics, Organizations, and Recipients using VF, HPoP, and VAMS	Conduct an online Salesforce survey	The target data collection period will be February 16, 2022 through February 15, 2023 A Salesforce online survey will be administered The survey will be programmed using Salesforce (survey tool) Individuals will receive an email (template in Appendix A) containing an invitation to participate in the survey, which contains a survey link. The invitation will indicate that the survey is voluntary, and that individual participant information will be kept confidential. Consent to participate will be assumed when the individual clicks on the provided link to complete the survey The anticipated response rate is 10% of those who receive the survey Survey results will be compiled and included in an evaluation report submitted to the CDC	Send emails containing request for input and link to online survey for users who contact the CARS HD	Collecting Dichotomous ('Yes/No') answers, free form responses and rated scaled response

1. Survey Invite Email Template

Hello {First name, Last name},

Thank you for contacting the CARS Help Desk. To continue providing the best service and customer experience, please assist us by taking a moment to complete a survey by sharing your experience with the CARS Help Desk. The survey is voluntary, and all information will be kept confidential.

{Provided Link}

Please be aware that consent is implied once the link has been accessed.

We appreciate your participation and will effectively use it to improve our services to you.

Thank you,

COVID-19 Administration Reporting Systems Help Desk (CARS)

Email: CARS_HelpDesk@cdc.gov

Phone: 1-833-748-1979

CONFIDENTIALITY NOTICE: This e-mail, including attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information or otherwise be protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender and destroy all copies and the original message.

Page 16 of 16