Medicare Part C and Part D Data Validation (42 CFR 422.516(g) and 423.514(g))

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

Medicare Part C and Part D Reporting

Requirements Data Validation Procedure Manual

Prepared by:

Centers for Medicare & Medicaid Services

Center for Medicare

Medicare Drug Benefit and C&D Data Group

Last Updated: February 2025

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0938-1115 (Expires: 12/31/2025). The time required to complete this information collection is estimated to average 30 hours per response, including the time to review instructions, search existing data resources, and gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.

Table of Contents

1. INTRODUCTION 1

1.1. Data Validation Requirement 1

1.2. DV Scope 1

1.3. Types of Organizations Required to Undergo DV 2

1.4. Required DV Manual and Tools 2

1.5. Organization of the Procedure Manual 3

EXHIBIT 1. DATA VALIDATION PROGRAM PHASES 3

EXHIBIT 2. DV PROGRAM ACTIVITIES 4

2. PART C AND PART D REPORTING SECTIONS REQUIRING DV 5

2.1. Parts C and D Reporting Sections Requiring DV 5

EXHIBIT 3. PART C AND PART D REPORTING SECTIONS REQUIRING DV 5

2.2. Reporting Sections Excluded from the DV Requirement 5

EXHIBIT 4. PART C AND PART D REPORTING SECTIONS EXCLUDED FROM DV 6

3. PLANNING FOR DV ACTIVITIES 6

3.1. Select appropriate contractor based on Standards for Selecting a DVC 6

3.1.1. Standards for Selecting a DVC 6

3.1.2. Timing of DVC Selection 6

3.1.3. Requesting a Contractor Change Mid-Review 7

3.2. Notify CMS of DVC Selection/Request Access to HPMS PRDVM 7

3.2.1. Documentation of DVC Selection Process 7

3.2.2. Request Access to HPMS PRDVM 7

3.2.2.1. PROCESS FOR OBTAINING HPMS ACCESS FOR SOS AND DVCS 7

3.2.2.2. PROCESS FOR SOS 7

3.2.2.3. PROCESS FOR DVCS 8

3.2.2.3.1. HPMS PRDVM Access 8

3.2.2.3.2. Instructions for Requesting Plan Access via EFI 8

3.2.2.3.3. Existing CMS Enterprise User Administration (EUA) User IDs and HPMS Access 8

3.2.2.3.4. Submission of Findings to CMS: Annual DV Review 8

3.3. Complete the Web-based DV Training 8

3.4. Review all DV Documents 9

3.4.1. Introduction to the EES 9

3.4.2. EES (Appendix B) 9

3.4.2.1. DV STANDARDS INSTRUCTIONS 9

3.4.2.2. DV STANDARDS 1 - 7 10

3.4.2.2.1. Standard 1 10

EXHIBIT 5. STANDARD 1: REQUIRED DATA FIELDS ARE ACCURATELY CAPTURED AND PROPERLY DOCUMENTED 10

3.4.2.2.2. Standard 2 10

3.4.2.2.3. Standard 3 10

3.4.2.2.4. Standards 4 and 5 10

3.4.2.2.5. Standards 6 and 7 10

4. PERFORMING DV ACTIVITIES 11

4.1. Complete OAI (Appendix E) and Provide Appropriate Documentation to Selected DVC per the OAI Documentation Request 11

4.2. Analyze OAI Responses 11

4.2.1. Perform OAI Gap Analysis 11

4.2.2. Review Source Code and Other Documentation 12

4.2.3. Prepare Interview Discussion Guide (IDG) (Appendix F) 12

4.3. Prepare for On-Site or Virtual Visit 12

4.3.1. Select Dates and Appropriate Location(s) of On-Site or Virtual Visit 12

4.3.2. Develop Agenda for Site Visit 12

4.3.3. Data Extraction and Sampling 12

4.4. Conduct Site Visit 13

4.4.1. Conduct Entrance Conference 13

4.4.2. Conduct Interviews with Organization Staff 13

4.4.3. Observe Reporting Processes 13

4.4.4. Conduct Exit Conference 14

4.5. Request Additional Documents (If Required) 14

5. ANALYZING RESULTS AND SUBMISSION OF FINDINGS 14

5.1. Reporting DV Results: How to Determine Scoring 14

5.2. The EES 14

5.2.1. Reporting Findings for Standards Using Binary Scale 14

EXHIBIT 6. EXAMPLES OF CALCULATIONS TO DETERMINE MINIMUM THRESHOLD OF CORRECT SAMPLE/CENSUS RECORDS

FOR “YES” FINDING 15

EXHIBIT 7. EXAMPLE OF HOW TO DETERMINE MINIMUM THRESHOLD OF IMPLEMENTED POLICIES OR PROCEDURES FOR

“YES” FINDING 15

5.2.2. Reporting Findings for Standards Using Likert Scale 15

EXHIBIT 8. EXAMPLE OF HOW TO DETERMINE MINIMUM THRESHOLD OF IMPLEMENTED POLICIES OR PROCEDURES FOR

FINDINGS USING BINARY AND LIKERT SCORE 16

5.2.3. Guidance for Interpreting Standards and Making a Findings Determination from Review of the EES 16

5.2.3.1. STANDARD 1 16

EXHIBIT 9. EXAMPLE ROWS FROM EES (APPENDIX B) AND GUIDANCE FOR STANDARD 1 17

5.2.3.2. STANDARD 2 17

5.2.3.3. STANDARD 3 20

EXHIBIT 11. EXAMPLE ROWS FROM EES (APPENDIX B) FOR STANDARD 3 20

5.2.3.4. STANDARD 4-7 21

EXHIBIT 12. EXAMPLE ROWS FROM EES (APPENDIX B) FOR STANDARDS 4 THROUGH 7 22

5.3. Submit DV Review Findings via HPMS PRDVM 23

5.3.1. Review Draft Findings with Sponsoring Organization 23

5.3.2. Submit DV Review Findings via HPMS PRDVM 23

6. Post-DV Activities 24

6.1. Compile Archive of DV Work Papers 24

EXHIBIT 13. MINIMUM DOCUMENTATION REQUIRED FOR DV ARCHIVE 24

6.2. Review Final DV Results 24

6.2.1. Pass/Not Pass Determination 24

6.2.2. Passing DV – Minimum threshold 25

6.2.3. SO Appeal of DV Determination 25

Medicare Part C and Part D Reporting Requirements Data Validation Procedure Manual

List of Appendices

• Appendix A: Standards for Selecting a Data Validation Contractor

• Appendix B: Examination Engagement Standards

• Appendix C: Model Language for Letter to Confirm Selection of Data Validation Contractor

• Appendix D: Example Application for Access to CMS Computer Systems

• Appendix E: Organizational Assessment Instrument

• Appendix F: Interview Discussion Guide

• Appendix G: Example Site Visit Agenda for On-site or Virtual Visits

• Appendix H: Data Extraction and Sampling Instructions

• Appendix I: Example Data File Inventory Log

• Appendix J: Data Validation Pass/Not Pass Determination Methodology

1. INTRODUCTION

   1. Data Validation Requirement

The Centers for Medicare & Medicaid Services (CMS) requires that sponsoring organizations (SOs) contracted to offer Medicare Part C and/or Part D benefits be subject to an independent yearly audit to validate certain data reported to CMS to determine its reliability, validity, completeness, and comparability in accordance with specifications developed by CMS.¹

Data validation (DV) allows CMS to use plan-reported data to:

• Respond to inquiries from Congress.

• Conduct oversight and provide information to agencies and the public about an SO’s performance using indicators such as operations, costs, availability and use of services.

The data allows CMS to:

• Monitor and compare the performance of SOs over time.

• Manage Star Ratings and other performance measures.

SOs can review the DV results to assess their performance and improve internal data, systems, and reporting processes. The Procedure Manual (Manual) provides SOs and data validation contractors (DVCs) with information about the Part C and Part D Reporting Requirements (RR) Data Validation program. The Manual describes:

• Background information.

• An overview of the DV program.

• Scope and timeframe required for DV.

• Tools and processes used for conducting DV.

2. DV Scope

CMS requires that SOs subject the plan-reported data to an independent DV once per year. DV will:

• Take place from April 1 – June 15.

• Incorporate all data submitted to CMS in the Health Plan Management System (HPMS) by the submission deadline (Exhibit 3) or the March 31 resubmission deadline for the reporting period. Should March 31 fall on a weekend or federal holiday, resubmissions must be submitted prior to 11:59 pm the preceding business day.

DVCs cannot audit any RR data an SO submits or re-submits in HPMS after March 31 or RR data not submitted in HPMS. The reviewer must submit findings from the annual DV review to CMS by June 15.

3. Types of Organizations Required to Undergo DV

CMS requires Part C and Part D SOs that report Part C and/or Part D data to CMS per the RR, regardless of enrollment size, to undergo DV. The only SOs exempt from participating in the DV are:

• Program of All-Inclusive Care for the Elderly (PACE) SOs and Part C Health Care Prepayment Plans (HCPPs).

• Employer Group Waiver Plans (EGWPs) and all-800 series plans are waived from the Enrollment and Disenrollment reporting section. For contracts with both non-800-series and 800-series plans, data for the 800-series plan(s) may be excluded from the Enrollment and Disenrollment reporting section.

• Organizations/sponsors that non-renew or terminate a contract during the measurement year.²

• A contract that terminates before July 1 in the following year after the contract year (CY) reporting period.³

Any SO that delegates the data collection, calculation, and/or reporting for any reporting section or data element must have the DVC include the delegated entity’s data and reporting processes in its DV review for each applicable contract. For example, CMS requires delegated entities to provide applicable policies, procedures, and source data to the DVC for validation if they submit data to an SO used for any reporting section.

4. Required DV Manual and Tools

DVCs must use this manual and its appendices to conduct DV. The manual includes the following documents:

1. Standards for Selecting a Data Validation Contractor (Appendix A).

2. Examination Engagement Standards (EES) (Appendix B).

3. Model Language for Letter to Confirm Selection of Data Validation Contractor (Appendix C).

4. Example Application for Access to CMS Computer Systems (Appendix D).

5. Organizational Assessment Instrument (OAI) (Appendix E).

6. Interview Discussion Guide (IDG) (Appendix F).

7. Example Site Visit Agenda for On-site or Virtual Visits (Appendix G).

8. Data Extraction and Sampling Instructions (Appendix H).

9. Example Data File Inventory Log (Appendix I).

10. Pass/Not Pass Determination Methodology (Appendix J).

The EES and other documentation associated with the DV program assess an SO’s information systems capabilities, and overall processes for collecting, storing, compiling, and reporting the data as specified by the Part C and Part D RR.

The most current versions appear at Medicare Part C and Part D Data Validation and within the HPMS Plan Reporting Data Validation Module (PRDVM). Before conducting DV, all SOs and DVCs should confirm that

they’re using the most recent DV documentation available on the CMS DV website.

5. Organization of the Procedure Manual

Exhibit 1 below illustrates the organization of the Manual. The four phases that comprise the DV process structure and the document’s content. The graphic shows the phases in the order in which the annual DV cycle is conducted.

EXHIBIT 1. DATA VALIDATION PROGRAM PHASES

Each phase of the DV review process contains several activities. Exhibit 2 shows the activities in order, beginning with the selection of an appropriate DVC and ending with the appeal of DV determinations.

During the DV process:

• The SO provides necessary documentation and information to the DVC.

• The DVC must serve as an independent reviewer and may not provide consultative services to the SO. Reference 3.1.1 for more information.

We describe each of these steps in more detail throughout the Manual.

EXHIBIT 2. DV PROGRAM ACTIVITIES

DV Phase	Step	Responsible Party	DV Activities	Timeline
Planning for DV Activities	1	SO	Select independent DVC based on Standards for Selecting a Data Validation Contractor (Appendix A).	DecemberMarch
	2	DVC, SO	Notify CMS of DVC Selection/Request Access to HPMS PRDVM. Enter/Update the DVC organization in HPMS. If applicable, enter/update the pre-assessment consultant in HPMS.	January-April
	3	DVC, SO	Complete the web-based Data Validation Training.	February-March
	4	DVC, SO	Review all DV documents.	January-March
Performing DV Activities	5	SO	Complete OAI (Appendix E) and provide appropriate documentation to selected DVC per the OAI’s documentation request.	March-April
	6	DVC, SO	Analyze OAI (Appendix E) Responses.	March or later
	7	DVC, SO	Prepare for on-site or virtual visit (site visit agenda, resource needs, and logistics).	Early April
	8	DVC, SO	Conduct on-site or virtual review (convene entrance conference, conduct interviews with SO staff, observe SO’s reporting processes, and obtain census and/or sample files).	Early April
	9	DVC	Request additional documents following on-site or virtual visit (if applicable).	Mid/Late April
Analyzing Results and Submission of Findings	10	DVC	Determine compliance with Data Validation Standards by recording findings in the EES (Appendix B).	April-June 15
	11	DVC	Submit EES findings to CMS via HPMS PRDVM and receive DV scores.	No later than June 15
Completing Post- Activities	12	DVC, SO	Compile archive of DV work papers.	No later than June 15
	13	DVC	Provide draft findings to the SO.	No later than June 15
	14	DVC, SO	Review EES findings and make necessary technical and factual corrections.	No later than June 15
	15	SO	Appeal DV determination(s) (if applicable).	5 business days following the DV deadline

* References to December refer to the calendar year before the DV review; all other references to months refer to the same calendar year as the DV review.

2. PART C AND PART D REPORTING SECTIONS REQUIRING DV

This section provides an overview of the Part C and Part D reporting sections that will undergo DV.

1. Parts C and D Reporting Sections Requiring DV

EXHIBIT 3. PART C AND PART D REPORTING SECTIONS REQUIRING DV

Reporting Section	Reporting Period(s)	Data Submission Due Date(s) to CMS
Part C
Grievances	1/1 -3/31 4/1 -6/30 7/1 -9/30 10/1 -12/31	First Monday of February
Organization Determinations/ Reconsiderations	1/1 -3/31 4/1 -6/30 7/1 -9/30 10/1 -12/31	Last Monday of February
Special Needs Plans (SNPs) Care Management	1/1 -12/31	Last Monday of February
Part D No data No data
Medication Therapy Management (MTM) Programs	1/1 -12/31	Last Monday of February
Grievances	1/1 -3/31 4/1 -6/30 7/1 -9/30 10/1 -12/31	First Monday of February
Coverage Determinations, Redeterminations (including AtRisk Redeterminations Under a Drug Management Program), and Reopenings	1/1 -3/31 4/1 -6/30 7/1 -9/30 10/1 -12/31	Last Monday of February
Improving Drug Utilization Controls	1/1 -3/31 1/1 -6/30 1/1 -9/30 1/1 -12/31	Last Monday of February

2. Reporting Sections Excluded from the DV Requirement

Exhibit 4 lists the reporting sections excluded from the DV review at this time.

EXHIBIT 4. PART C AND PART D REPORTING SECTIONS EXCLUDED FROM DV

	Part C Reporting Sections		Part D Reporting Sections
• • • • •	D-SNP Enrollee Advisory Committee D-SNP Transmission of Admission Notifications Enrollment/Disenrollment Employer Group Plan Sponsors Rewards and Incentives Programs Payments to Providers Supplemental Benefit Utilization and Costs	• • •	Enrollment/Disenrollment Employer/Union-Sponsored Group Health Plan Sponsors Medicare Prescription Payment Plan

3. PLANNING FOR DV ACTIVITIES

   1. Select appropriate contractor based on Standards for Selecting a DVC

The SO is required to select an independent, external DVC to complete DV and report the findings to CMS.

1. Standards for Selecting a DVC

CMS provides a set of Standards for Selecting a DVC (Appendix A) for SOs to use in acquiring a contractor. The selected DVC must have these standards, which describe the minimum: • Qualifications,

• Credentials,

• Resources, and

• Conduct guidelines.

SOs must acquire one DVC to conduct the DV and, if necessary, the DVC may subcontract to ensure that it has the expertise required for each DV area and to meet the minimum standards. SOs may use their own staff only to assist the DVC in obtaining the information, data, and documents needed to complete the DV review.

The Standards for Selecting a DVC also contain best practices that CMS expects DVCs to follow during the review. CMS requires the DVC to:

• Remain objective and independent in all its activities.

• Keep the SOs’ privileged information confidential.

The DVC may not perform mock audits, pre-assessments, and any other types of review before the DV review period from April-June. However, the DVC can start preparing for the DV review before April 1 so that DV can start as soon as possible on or after April 1. Examples of preparation activities may include:

• Meeting with the SO to discuss the DV process, resource needs, timeline, etc.

• Providing the SO with a list of documents, data, and materials needed to complete the review.

Any specific questions regarding organizational independence should be directed to PartCandD_Data_Validation@cms.hhs.gov.

2. Timing of DVC Selection

An SO must select a DVC before the beginning of the DV period. SOs should allow enough time for the

DVC to perform all the requirements of the review and submit findings to CMS via the PRDVM in HPMS during the required timeframe.

3. Requesting a Contractor Change Mid-Review

An SO may not change its DVC during the formal review period from April 1 to June 15 unless there are conditions that are unrelated to DV findings such as negligence or malfeasance on the part of the DVC. If an SO needs to change its contractor, CMS requires the new DVC to complete the DV review in its entirety, starting with the OAI (Appendix E) analysis and through the submission of findings to CMS by June 15.

2. Notify CMS of DVC Selection/Request Access to HPMS PRDVM

   1. Documentation of DVC Selection Process

SOs must document their DVC selection process and be able to show, upon CMS’ request, how their chosen DVC meets the minimum qualifications, credentials, and resources described in the Standards for Selecting a DVC. This includes maintaining a copy of the documentation that all DVC staff assigned to the applicable DV review team completed the CMS Data Validation Training program (see Section 3.3). The SO must retain this documentation for 10 years as required under 42 CFR §§ 422.504(d) and 423.505(d). DVC staff must complete this training each calendar year.

If an SO chooses the same DVC it used for a previous year’s DV review, it must still document the selection process as described above.

CMS does not require the SO to submit this information to CMS, but the information should be available if CMS requests it.

2. Request Access to HPMS PRDVM

Once the SO selects a DVC and documents the selection process, the DVC requests staff access to the PRDVM in HPMS. This module allows users to gain access to pertinent documents, record DV results, and submit findings to CMS. The credentials CMS assigns to a user will allow that individual to access only those associated SO(s)/contract(s). The DVC staff will use these credentials to access the appropriate documents within the PRDVM starting no earlier than April 1 of the calendar year.

1. PROCESS FOR OBTAINING HPMS ACCESS FOR SOS AND DVCS

CMS requires users who access the HPMS to have a user ID. Send any questions you have regarding the user ID process to HPMS_access@cms.hhs.gov. Read more about obtaining HPMS access at the CMS HPMS website.

2. PROCESS FOR SOS

To gain access to the PRDVM, the SO sends their DVC an official letter in either hardcopy or an emailed PDF format attachment with the following information:

• The SO’s acknowledgment that it has contracted with the selected DVC to complete the review,

• The name of each individual who needs access (up to 5 users),

• The type of functionality that each user requires,

• Acknowledgement that the users have completed the web-based DV Training,

• The contract number(s) the DVC will need access to, and

• The SO’s Chief Executive Officer’s (CEO) signature.

You can find model language for this letter in the Model Language for Letter to Confirm Selection of a Data Validation Contractor (Appendix C).

If an SO chooses the same DVC it used for a previous year’s DV review, it must still provide the DVC with this signed letter for the current year’s DV activities.

3. PROCESS FOR DVCS

   1. HPMS PRDVM Access

DVC staff must get individual access to the HPMS PRDVM. If the designated user(s) from the DVC does not have active access to HPMS, each user should download the Application for Access to CMS Computer Systems at: Access to CMS Data & Application and follow the instructions provided in Example Application for Access to CMS Computer Systems (Appendix D) for requesting reviewer access to the HPMS PRDVM.

CMS allows up to 5 users from each DVC to have access to this Module on behalf of each SO. The DVC must create their user ID using EFI and then email the letter from each SO for which they are under contract to complete the DV review. Since CMS will process the request for a new user ID first, the user will only get access to the HPMS main home page until we receive a DV letter. You can send the letters as email attachments to HPMSConsultantAccess@cms.hhs.gov.

2. Instructions for Requesting Plan Access via EFI

Follow the instructions for getting a user ID by clicking the Instructions for Requesting Plan Access via EFI link under the Download section of the CMS HPMS website.

When the user gets to the “I am a” question, the user must select Data Validation Consultant from the drop down. The system will automatically fill out the company name and Plan number. The HPMS team created an instructional video to walk a user through getting a new CMS user ID.

3. Existing CMS Enterprise User Administration (EUA) User IDs and HPMS Access

Users who already have active CMS Enterprise User Administration (EUA) User IDs and HPMS access do not need a new Application for Access to CMS Computer Systems. Instead, they must modify their current credentials to allow access to the PRDVM. For this access, users need to ensure that the letter from each SO linking the DVC to the SO includes the user’s current User ID and an explanation that the user already has HPMS Access. Users must send this letter to CMS via email to HPMSConsultantAccess@cms.hhs.gov. If a user had PRDVM access as a DVC previously, they do not need to resend letters on their behalf. CMS only requires letters for changes to a DVC user’s account.

4. Submission of Findings to CMS: Annual DV Review

DVCs must submit the findings from the annual DV review to CMS by June 15 of each calendar year. To assure timely access to the HPMS PRDVM to meet this DV timeframe, submit requests for HPMS PRDVM access by early April. It will take about 4 weeks for the designated individuals to get the credentials (CMS EUA User IDs and passwords) to access the PRDVM.

3. Complete the Web-based DV Training

CMS developed a web-based DV Training for SOs and DVCs to learn more about the DV program and its specific requirements.

During the DV preparation phase, CMS recommends that all SO staff involved in DV complete the CMS webbased DV Training to familiarize themselves with the DV process and requirements.

Also, CMS requires all DVC staff to take the CMS web-based DV Training prior to working on the DV project. They must complete this training each year. Once they complete training, they will receive a generated certificate of completion. DVC staff should provide training certificates to the SO before starting work on the DV.

4. Review all DV Documents

Refer to Section 1.4 for the documents you need to review before DV. This section will focus specifically on the DV standards, which the EES (Appendix B) further describes.

1. Introduction to the EES

The DV Standards include general and elemental standards that DVC staff must use to determine whether the data each SO reported to CMS per the Part C and Part D RR and Technical Specifications (TS) are reliable, valid, complete, and comparable in accordance with the specifications developed by CMS.

The standards assess an SO’s information systems capabilities and its processes for collecting, storing, compiling, and reporting Part C and/or Part D data. They also assess whether SOs follow the applicable RR and TS to compile data, consider appropriate data exclusions, and verify calculations, computer code, and algorithms.

2. EES (Appendix B)

The DV Standards include general and elemental standards that DVC staff must use to determine whether the data each SO reported to CMS per the Part C and Part D RR and TS are reliable, valid, complete, and comparable in accordance with the specifications developed by CMS.

The standards assess an SO’s information systems capabilities and its processes for collecting, storing, compiling, and reporting Part C and/or Part D data. They also assess whether SOs follow the applicable RR and TS to compile data, consider appropriate data exclusions, and verify calculations, computer code, and algorithms.

1. DV STANDARDS INSTRUCTIONS

The DV Standards include identical instructions relating to the types of information that DVCs must review for each reporting section and a set of validation standards based on the applicable RR and TS.

The DVC uses these standards with the Data Extraction and Sampling Instructions (Appendix H) and the EES (Appendix B) to evaluate the SO’s processes for producing and reporting the reporting section’s data. CMS strongly recommends that the DVC and SO staff responsible for RR and/or DV review the DV Standards documentation before and during the review of each reporting section to ensure that they thoroughly understand the standards. This will also help to ensure DVCs extract all applicable data fields for each reporting section.

The top portion of each set of standards details the documents and reports what the DVC uses to determine compliance with the standards for each specific reporting section. The gray box underneath the name of the applicable reporting section lists the documents and reports.

This section also contains notes to the DVC regarding a specific reporting section, if applicable, and any nuances or differences that DVCs may encounter during the review of that reporting section. The second section of each set of standards is identical for all Part C and Part D reporting sections.

2. DV STANDARDS 1 - 7

   1. Standard 1

Standard 1 (see Exhibit 5) provides general and specific criteria for DVCs to review the database management and structure of source documentation used by the SO.

EXHIBIT 5. STANDARD 1: REQUIRED DATA FIELDS ARE ACCURATELY CAPTURED AND PROPERLY DOCUMENTED

DV STANDARD 1

1. Database management and structure: A review of source documents (for example, programming code, spreadsheet formulas, analysis plans, saved data queries, file layouts, process flows) shows that all source documents accurately capture required data fields, and the SO has properly documented them.

Criteria for Validating Source Documents:

1. SOs properly secured source documents and output and have referenced data file locations correctly so that DVCs can retrieve source documents at any time to validate the information submitted to CMS via CMS systems.

2. SOs clearly and adequately documented source documents.

3. SOs accurately captured required data fields including all data fields for RR in source documents. All data fields have meaningful, consistent labels (for example, label field for patient ID as Patient ID, rather than Field1 and maintain the same field name across data sets).

4. Source documents are error-free (for example, programming code and spreadsheet formulas have no messages or warnings indicating errors).

5. SOs appropriately applied version control of source documents.

6. SOs captured the appropriate date range(s) for the reporting period(s). SO reports data based on the required reporting period of 1/1-12/31.

7. SOs assigned data at the applicable level (for example, plan benefit package (PBP) or contract level).

2. Standard 2

Standard 2 instructs the DVC to review data base extraction functions and validate the underlying source data, preliminary data sets, and interim data sets used by the SO to populate each reporting section. Standard 2 requires that the population for each reporting section are accurately identified, processed, and verified (including calculations for the number of enrollees, claims, grievances, procedures, etc.). The DVC must review that QA checks/thresholds are applied by the SO to detect outlier or erroneous data.

3. Standard 3

DVCs use Standard 3 to determine whether an SO implements policies and procedures for submission of their final stage data into HPMS. The DVC validates that each data element for each reporting section is valid, complete, and accurate by reviewing expected counts and that the SO accurately captures data by applying data integrity/ logical checks.

4. Standards 4 and 5

For Standards 4 and 5, the DVC must verify that the SO implements policies and procedures for regular data system updates, archiving of source, intermediate, and final stage data, and restoration. The Standards ensure that SOs keep data systems up to date and that they have systems in place for accurate and timely data submission or re-submission in the event of data loss.

5. Standards 6 and 7

DVCs apply Standards 6 and 7 only in certain situations. Standard 6 is applicable if an SO’s data

systems underwent any changes during the reporting period. If so, the DVC must examine and confirm whether the changes were properly implemented and that they did not adversely impact the reported data.

Standard 7 applies if the SO outsourced any of the data collection or validation processes to another entity. This standard assesses whether the SO has policies and procedures in place to routinely monitor the delegated entity’s work to ensure quality and timeliness of the data.

The DVC should mark “Not Applicable” in EES (Appendix B) if Standard 6 or 7 is not applicable to the reporting section or contract under review.

4. PERFORMING DV ACTIVITIES

   1. Complete OAI (Appendix E) and Provide Appropriate Documentation to Selected DVC per the OAI Documentation Request

The OAI (Appendix E) focuses on how the SO collects, stores, and reports data. We recommend SOs complete the OAI before the DV, as the DV review relies significantly on the information in this tool. CMS estimates that the OAI should take a minimum of two weeks to complete and submit it to the DVC no later than early April. SOs may not send their completed OAI or source code, SOPs, etc. to their DVCs before the start of the DV cycle on April 1.

CMS requires each SO to provide its DVC with the basic information of its Medicare contracts and which Part C and/or Part D reporting sections each contract submits to CMS. SOs with more than one contract with CMS only need to complete one version of the OAI that covers all its contracts. If the SO varies the information they provide in the OAI allows for the flexibility to identify the differences for the DVC in applicable sections.

The SO should reflect its systems and processes in place during the reporting period(s) undergoing the DV review in all documentation and responses to questions in the OAI. For example, if the data CMS reviews are for the 2025 reporting period, the SO should include only diagrams of the information systems in place in 2025 or the programming code they used in 2025 to calculate the reporting sections, despite DV occurring in 2026.

The SO and its DVC must determine mutually agreeable methods for sharing and protecting proprietary data, such as data requested in the OAI, and protected health information. The Standards for Selecting a Data Validation Contractor (Appendix A) includes minimum security requirements, and the DVC’s facility, equipment, and processes must comply with those requirements. CMS requires the SO to ensure that the DVC complies with all Health Insurance Portability and Accountability Act (HIPAA) privacy and security requirements.

2. Analyze OAI Responses

CMS recommends DVCs perform a preliminary review of the documentation submitted in the OAI in advance of each site visit (on-site or virtual) so that DVCs can follow-up with any documentation during the site visit. CMS requires the documentation submitted by the SO be adequate for an effective review. The amount of detail SOs provide in the documentation determines the ease of the review process, especially for the review of programming code/source code.

Additionally, the OAI provides supplemental questions to help the DVC better understand the processes the SO uses to compile and submit its reporting sections. The SO’s responses to these questions will provide insight as to who is responsible for the quality control and submission of the data, the processes for incorporating CMS updates to the RR and/or TS into the SO’s systems, and descriptions of any past issues during the reporting process.

1. Perform OAI Gap Analysis

When the DVC receives the completed OAI, the DVC should review the document for completeness and accuracy. The DVC should note the sections of the OAI that the SO missed or did not complete, and the DVC should follow-up with the SO to complete these sections. CMS requires the DVC to determine whether any identified gaps in the OAI responses require the SO to address them before or during the site visit portion of the review.

2. Review Source Code and Other Documentation

Data dictionaries and source codes are critical for allowing the DVCs to map ambiguous field names and internal status codes to meaningful descriptions. Well organized and structured documentation of the reporting and data extraction processes for the various reporting sections will assist the DVC in gaining a more thorough understanding of the SO. The DVC should be familiar with data systems and processes detailed by the SO in the OAI to ensure thorough preparation for the site visit.

3. Prepare Interview Discussion Guide (IDG) (Appendix F)

The Interview Discussion Guide (IDG) (Appendix F) is intended to help the discussion between the DVC and the SO’s report owners and subject matter experts. The IDG is a dynamic tool that has both general and reporting section questions that can guide an effective discussion about an SO’s underlying data systems and reporting processes. If the DVC discovers evidence that may indicate errors in the SO’s data or reporting processes during review of the documentation the SO provided in response to the OAI, the DVC should change the IDG used for that SO with new questions that may show any vulnerabilities or opportunities for repeated errors with data collection or reporting. Additionally, the IDG should serve as a “guide” for the DVC; it is at the DVC’s discretion to include additional questions and/or details in the document to discuss during on-site or virtual interviews and to ensure they document the additional details accordingly.

3. Prepare for On-Site or Virtual Visit

   1. Select Dates and Appropriate Location(s) of On-Site or Virtual Visit

Site visits may be on-site or virtual. The DVC determines the most appropriate location(s) of the site visit

(for example, virtual, SO’s facility, PBM’s facility, other delegated entity’s facility). During the site visit, the DVC: (1) conducts interviews with SO staff, (2) observes the SO’s reporting processes, and (3) obtains census and/or sample files to support the validation of Part C and Part D reporting sections. SOs and DVCs must determine mutually agreeable dates for performing the site visit or holding a virtual site meeting.

2. Develop Agenda for Site Visit

CMS expects the DVC and SO to work together to prepare a site visit agenda. The time required to complete a site visit may be contingent upon the size of the SO, efficiency of the SO’s operations, level of reporting automation, and scope of the DV review. CMS requires the SO’s report owner(s) to schedule a DVC session for each reporting section to allow enough time for the SO to provide an overview of each of the relevant data systems they used gathering data and producing reports, as well as to complete the data extraction/sampling process.

3. Data Extraction and Sampling

CMS urges the DVC to review the completed OAI and, if necessary, hold conference calls with the SO to discuss the SO’s processes. Calls with the SO, specifically with each reporting section’s report owner, can also provide an opportunity for the DVC to review the Data Extraction and Sampling Instructions (Appendix H) in more detail and for the report owners to seek clarification as needed. These discussions can also inform the DVC about the SO’s data systems and the sources from which the SO pulls data.

The DVC can use two methods to extract data for each reporting section. The first is to extract the full census of data for a reporting section, meaning they extract every data record that is relevant to a reporting section. When possible, DVCs should attempt to extract the full census. The DVC extracting the census will enable them to determine with the greatest precision whether SOs submitted reporting sections accurately in HPMS. DVCs can use the second method of extracting a random sample, if the size or complexity of a database creates an unusual time burden on the DVC or SO, or both.

DVCs should use their best judgment to decide if extracting a full census is feasible, or if selecting a random sample will provide the data necessary for the DV review. Refer to Appendix H for further details regarding these two methods. CMS recommends that the DVC records details about each reporting section’s data set into a Data File Inventory Log (Appendix I). Appendix I contains an example log that the DVC can use. It includes details such as the reporting section name, report owner, data file name, type of data file (for example, source, intermediate, or final stage data file), number of rows or records, and a description of the file. By completing this log, the DVC will be able to easily reference the data files during its post-site visit assessment of data.

The SO should write all data files to tab-delimited or comma-delimited text files with variable names in the first row and transfer these files to the DVC’s secure storage device for each reporting section’s data. The SO must also provide the DVC a file layout or data dictionary for the data files in either Word documents or Excel spreadsheets on the same secure storage device. The SO and DVC must ensure that they have established mutually agreeable methods for sharing protected health information and that the DVC complies with all HIPAA privacy and security requirements.

4. Conduct Site Visit

   1. Conduct Entrance Conference

The entrance conference provides an opportunity for the DVC and the SO’s management and individual report owners to introduce themselves and discuss expectations for the on-site or virtual visit. At the entrance conference, the DVC should describe the objectives for the review and discuss any administrative needs of the team. Optionally, the SO may provide a high-level overview of its organization, focusing on its operations with respect to meeting the Parts C and D RR.

2. Conduct Interviews with Organization Staff

The DVC must conduct interviews with the subject matter experts and data owners for each reporting section. These interviews provide a first-hand opportunity for the DVC to gain a thorough understanding of each SO’s data collection and reporting processes with meeting the Parts C and D RR. The DVC should reference the IDG to ensure that they address all key topics during the interviews. In addition, they should address and identify any questions and follow-up items during the analysis of the OAI responses during the interviews.

3. Observe Reporting Processes

Designated SO staff (such as report owners) must provide demonstrations of the data systems and reporting processes including data extraction from originating data sources, data analysis, quality assurance processes, and processes for entering or uploading final data into CMS systems. The following is a sample list of the parts of the process that they should demonstrate:

• Contact information and location of report owner and data providers,

• Function and location of all data warehouses,

• Types of data used (format, number of tables),

• Links and joins to other areas/departments/data,

• Types of programming used to create the reports,

• Review processes and oversight,

• Timeframes for the process (amount of time it takes to run specific parts of the report),

• Approximations of report volume,

• Updates to the process and system changes, and

• Storage locations (for example, building or server), security, and access constraints.

The demonstrations provide a clear illustration of the reporting processes, provide the DVC with insight into the SO’s ability to ensure reliable, valid, complete, and comparable data, and allow an opportunity to get immediate responses to any questions or concerns about the reported data.

4. Conduct Exit Conference

The on-site or virtual visit should conclude with an exit conference, where the DVC should provide the SO with a summary of next steps and note any follow-up that may need to occur.

5. Request Additional Documents (If Required)

CMS recognizes that it may not be possible for DVCs to get all the required data and documentation during the scheduled on-site or virtual visit and follow-up conversations may be required. The DVC should make every attempt to gather all required data and documentation during the on-site or virtual visit. If they need to follow-up after the conclusion of the scheduled on-site or virtual visit due to not enough information, the DVC should have more conversations with the SO and/or make requests for documentation. DVCs and SOs should understand that the DV is an iterative process, and SOs should be ready to provide more data and documentation after the on-site or virtual visit.

5. ANALYZING RESULTS AND SUBMISSION OF FINDINGS

   1. Reporting DV Results: How to Determine Scoring

After the site visit, the DVC assesses the documentation and census/sample data from the SO, as well as the information they gained during the interviews and demonstrations of the SO’s reporting processes and information systems. The DVC determines compliance with the DV standards and records findings in the EES. They must score and record DV of all reporting sections using the EES.

2. The EES

In the scope of review, the DVC completes the EES (Appendix B) to record the findings for each contract and uploads as a data file into the PRDVM. The EES (Appendix B) allows the DVC to record notes, data sources they referenced, and findings for the different standards for a given reporting section. The DVC will record any reporting section-level and data element-level findings for each reporting section. After the DVC has submitted in HPMS, the SO can view the findings via the “Review Data Validation Findings Report”.

When using the EES (Appendix B), DVCs should only complete areas displayed in white for data sources, review results, and findings. Areas in gray are not applicable for scoring. In the "Data Sources and Review Results" column, the DVC should enter the data sources and review results for each standard or sub-standard. Next to this column, in the "Findings" column, the DVC selects the appropriate choice based on whether the plan met the requirements for the standard or sub-standard.

1. Reporting Findings for Standards Using Binary Scale

For all standards except 1.b, 1.c, 1.d, 2.a, 3.a and 3.b, DVCs report the findings using a binary scale. The DVC selects "Y" if the requirements for the standard or sub-standard have been completely met. The DVC must select "N" if any requirement for the standard or sub-standard was not met.

CMS expects that there will be situations when the DVC finds that an SO is not fully compliant with specific

DV standards. CMS has established a threshold where a minimum of 90 percent of records are accurate (for example, sample or census records, source documents, policies and procedures, data entry records) to record a “Yes” finding for any standard. DVCs apply this threshold to standards that require the review of policies and procedures when possible to quantify the adherence to or implementation of policies and procedures (see Exhibit 6). Exhibit 7 shows examples of how to calculate this minimum threshold specifically for Standard 1 e., where the DV involves samples or the complete census of records and/or data values.

EXHIBIT 6. EXAMPLES OF CALCULATIONS TO DETERMINE MINIMUM THRESHOLD OF CORRECT SAMPLE/CENSUS

RECORDS FOR “YES” FINDING

Sample/ Census Size	Calculation for Minimum Threshold	Minimum Threshold of Correct Records for “Yes” Finding
150	0.90 x 150 = 135	At least 135 of the records are correct for the reviewer to record “Yes” to the standard.
205	0.90 x 205 = 184.5	At least 185 of the records are correct for the reviewer to record “Yes” to the standard (round 184.5 to 185).

EXHIBIT 7. EXAMPLE OF HOW TO DETERMINE MINIMUM THRESHOLD OF IMPLEMENTED POLICIES OR PROCEDURES

FOR “YES” FINDING

Standard	Standard Description	Minimum Threshold for a “Yes” Finding
1 e.	Version control of source documents is appropriately applied.	SO appropriately applies version control of source documents. Eleven out of the 12 months in the contract year, the SO implemented the enrollment system update policy as written (11/12 = 91.6 percent).

2. Reporting Findings for Standards Using Likert Scale

For standards 1.b, 1.c, 1.d, 2.a, 3.a and 3.b, the scoring uses a five-point Likert-type scale. DVCs review the percentage of records that meet the standards, and enter a score based on the Likert- type scale. The scale corresponds to the percentage of errors that are found in plan data as shown below:

1. Contract data has more than 20 percent error in records – Contract receives a score of 1 for the given standard/sub-standard.

2. Contract data has between 15.1 percent and 20 percent error in records – Contract receives a score of 2 for the given standard/sub-standard.

3. Contract data has between 10.1 percent and 15 percent error in records – Contract receives a score of 3 for the given standard/sub-standard.

4. Contract data has between 5.1 percent and 10 percent error in records – Contract receives a score of 4 for the given standard/sub-standard.

5. Contract data has less than or equal to a 5 percent error in records – Contract receives a score of 5 for the given standard/sub-standard.

Exhibit 8 shows examples of different scenarios a DVC might face and corresponding scores DVCs assign to the contract for the standards using either Likert or binary scoring.

EXHIBIT 8. EXAMPLE OF HOW TO DETERMINE MINIMUM THRESHOLD OF IMPLEMENTED POLICIES OR PROCEDURES

FOR FINDINGS USING BINARY AND LIKERT SCORE

Standard4	Percentage of Errors That the DVC Found in Plan Data	DV Response (DV Findings the DVC Reported on Column 'F' in Appendix B)
1.a, 1.e, 1.f, 1.g, 4, 5, 6, 7	Fewer than 10 percent error in contract data for the given reporting section/ data element(s)	Yes
1.a, 1.e, 1.f, 1.g, 4, 5, 6, 7	More than 10 percent error in contract data for the given reporting section/ data element(s)	No
6, 7	If any of the listed standards is not applicable	N/A
Standard	Percentage of Errors That the DVC Found in Plan Data	DV Response (DV Findings the DVC Reported on Column 'F' in Appendix B)
1.b, 1.c, 1.d, 2.a, 3.a, 3.b	More than 20 percent error in contract data for the given reporting section/data element(s)	1
1.b, 1.c, 1.d, 2.a, 3.a, 3.b	Between 15.1 percent and 20 percent error in contract data for the given reporting section/data element(s)	2
1.b, 1.c, 1.d, 2.a, 3.a, 3.b	Between 10.1 percent and 15 percent error in contract data for the given reporting section/data element(s)	3
1.b, 1.c, 1.d, 2.a, 3.a, 3.b	Between 5.1 percent and 10 percent error in contract data for the given reporting section/data element(s)	4
1.b, 1.c, 1.d, 2.a, 3.a, 3.b	5 percent error or less in contract data for the given reporting section/data element(s)	5

3. Guidance for Interpreting Standards and Making a Findings Determination from Review of the EES

To ensure consistency with the review process, CMS lists these descriptions of the data sources and criteria that DVCs use to determine findings for each of the DV standards/sub-standard.

1. STANDARD 1

DVCs assess this validation standard at the reporting section level and use it to determine that all source documents accurately capture required data fields and the SOs properly documented them.

Exhibit 9 describes the guidance for evaluation Standard 1, which illustrates an example of the EES (Appendix B) and how to evaluate Standard 1. The DVC will assess this standard at the reporting section level and determine a finding for each of the seven sub-standards contained in

Standard 1.

EXHIBIT 9. EXAMPLE ROWS FROM EES (APPENDIX B) AND GUIDANCE FOR STANDARD 1

Standard/ Substandard ID	Standard/Sub-standard Description	Guidance
1	Database management and structure: • A review of source documents (for example, programming code, spreadsheet formulas, analysis plans, saved data queries, file layouts, process flows) shows that all source documents accurately capture required data fields, and the SO has properly documented them. .	Determine if the SO’s source documents (for example, programming code, spreadsheet formulas, analysis plans, saved data queries, file layouts, process flows) accurately capture the data fields required for each reporting section under review and are documented with the necessary detail and information to create data file sets and other outputs. Ensure that all source documentation is legible, descriptive, and understandable, including each of the following: Standard Operating Procedures (SOPs) include detailed workflows and processes related to managing, producing, and tracking source documents. Titles and footnotes SOs used in programs and the reported output is legible and correspond to HPMS reports and tables. If used, they properly documented macros. SOPs, file-naming conventions, dates of source documents and output reports reflect application of version control. SOs reference data file locations correctly within source code (such as, these files can be located using the references that exist within the source code). Dated HPMS entries match the source document(s) to create the data entered in the HPMS. Verify that SOs selected data at the proper level (for example, either the contract or the PBP level). Check date ranges, demographic information, and eligibility information to examine proper data filtering. Ensure that the DVC uses the current documentation, and it is relevant to the time period of the RR.

Please note that DVCs should address Standards 1 and 2 concurrently since the evaluation of source documents directly impacts the quality of the underlying data and vice versa so that DVCs accurately identify, process, and calculate elements for each reporting section. For example, the DVC should ensure that all source documentation (file layouts, data dictionaries, programming code, work instructions, SOPs, etc.) is available, and it allows for the complete validation of each reporting section.

2. STANDARD 2

DVCs assess whether SOs accurately identified, processed, and verified (including calculations for the number of enrollees, claims, grievances, procedures, etc.) the source document’s data populated for each reporting section for this validation standard. Each DVC should ensure that it has staff fluent in the SO’s programming language (i.e., SQL, SAS, or Microsoft VBA). Standard 2 requires the DVC to assess reporting section-level findings for Sub-Standard 2.a. Exhibit 10 illustrates an example of the EES (Appendix B) and guidance for Standard 2, Sub-Standard 2.a.

Since the DVC conducts the DV reviews at the contract level, for the reporting sections that require reporting at the PBP level, if the DVC finds that the SO incorrectly identified, processed, or calculated the data reported for any of the PBPs included under a contract, then the DVC must assign a 1, 2, or 3 finding in the EES (Appendix B) for the entire contract for the applicable substandard or data element (for Sub-Standard 2.a).

Exhibit 10 provides several examples of how to review source codes and evaluate the integrity of the data. However, the DVC may use other methods of DV to ensure a comprehensive and complete review of the source codes and census/sample data. The DVC must clearly document all errors they found in programming codes, referring to the program examined, the precise location in the program, the nature of the error, and the impact of the error in the “Data Sources and Review Results” section of the EES (Appendix B). Likewise, they must clearly document in the applicable section of the EES (Appendix B) any evidence from the review of census/sample data that leads to a negative finding.

While DVCs should carefully inspect the source codes to detect most errors and outliers in the reported data, they should conduct a careful review of the census or sample data gathered from the SO to minimize the chance that they didn’t detect a programming error. DVCs can check many of the same items as they review the source codes and analyze the extracted data sets. The DVC must also review that QA checks/thresholds are applied by the SO to detect outlier or erroneous data.

EXHIBIT 10. EXAMPLE OF THE EES (APPENDIX B) AND GUIDANCE FOR STANDARD 2, SUB-STANDARDS 2.A

Standard/ Sub- standard ID	Standard/Sub-standard Description	Guidance
2	Database Extraction Function: • Review of source data, preliminary data sets, and interim data sets (for example, programming code, spreadsheet formulas, analysis plans, saved data queries, file layouts, process flows) indicates the following: SOs accurately identify, process, and verify the population for each reporting section (including calculations for the number of enrollees, claims, grievances, procedures, etc.). They apply QA checks/thresholds to detect outlier or erroneous data.	Assess the programming code to determine if SOs extracted the data from the system properly and if the calculations SOs used in reporting data to CMS are accurate according to the reporting section applicable under review. A thorough review of source codes must examine every line of code to ensure the following for each reporting section under review: Data are extracted from the appropriate source system: Verify that all data sets found in the programming code can be traced back to the appropriate source data sets. SOs filtered data sets correctly: Verify that SOs applied data inclusion and exclusion criteria according to the reporting section criteria. For example, proper inclusion of records would ensure that source code indicates that only those records falling within the reporting period date range are included in the reported data. An example of correct exclusion would document source code that indicates beneficiaries are not eligible for a particular benefit (for example, Medication Therapy Management Program). Individual data sets are joined or merged correctly (this is especially important when moving data from source data sets to intermediate data sets). Verify that SOs used the correct key data field to generate the new data set and that they used the correct type of join (or data merge) to avoid creating duplicate records or improperly combining records from various data sets. Data set progression is accurate: Verify that required data fields in both the source and final stage files allow for file comparison and understanding of data production from source system through the final stage file. o If full census data is not extracted, verify that the sample size is sufficient and representative of the population of interest. While the Data Extraction and Sampling Instructions provide minimum sample sizes, DVCs often will need larger data sets to check for errors that occur infrequently. Statisticians should rely on standard statistical practices when determining the proper sample sizes that any estimates generated are statistically significant. All preliminary data sets and interim data sets are accurate: Verify that each data set is consistent with the standard/sub-standard description.

3. STANDARD 3

Standard 3 contains two Sub-Standards. Sub-Standard 3.a and 3.b requires the DVC to assess data element-level findings which examines each data element at their final stage for compliance with the data elements reported into HPMS by the SO. In addition, DVCs assess Standard 3 at the data element-level for reporting sections that are manually entered into the HPMS Plan Reporting Module (PRM) because it confirms that there were no manual data entry errors for each data element and that the SO accurately captures data by applying data integrity/logical checks, and for reporting sections reported as file uploads, it confirms that the SO used the correct file layout. Exhibit 11 shows an example of the EES (Appendix B) and guidance for evaluating Standard 3.

EXHIBIT 11. EXAMPLE ROWS FROM EES (APPENDIX B) FOR STANDARD 3

Standard/ Sub- standard ID	Standard/Sub-standard Description	Guidance
3	SOs implements policies and procedures in their final stage data sets for submission into HPMS, including the following: Expected counts - Data elements are valid, complete, and accurate according to the source document that SOs use to upload/enter data into the HPMS; ranges of data fields are verified; all calculations (for example, derived data fields) are verified; they properly address missing data; reporting output matches corresponding source documents (for example, programming code, saved queries, analysis plans); they appropriately apply version control of reported data elements. SOs accurately captures data by applying data integrity/logical checks; They apply QA checks/thresholds to detect outlier or erroneous data.	Determine who is responsible for entering/uploading data into CMS systems for each reporting section under review and if the SO wrote work instructions or policies and procedures for the entry or submission of the Part C and Part D RR. Evaluate Sub-Standard 3.a and 3.b by performing the following actions: Assess the Submission Activity Report from the HPMS PRM by performing the following: Request a copy of the contract’s Submission Activity Report from the SO: This report displays information about the original submission and all subsequent resubmissions for a particular contract or contracts. The report also displays Reporting Period, Contract Number, Plan ID, Submission Version, Due Date, and Date Submitted for each section. For either of the above scenarios, the DVC must clearly document the circumstances in the “Data Sources and Review Results” section of the EES (Appendix B). Assess the census/sample data provided by the SO to determine each of the following for each reporting section under review: SOs selected data records properly: Perform frequency calculations to list all unique occurrences of data fields pertinent to the calculation of the data element to verify they contain values within an acceptable range for the data field. Calculating frequency of occurrence for certain data fields might also alert the DVC to obvious mistakes in the data extraction.

EXHIBIT 11. EXAMPLE ROWS FROM EES (APPENDIX B) FOR STANDARD 3 (CONT.)

Standard/ Sub- standard ID	Standard/Sub-standard Description	Guidance
3		SOs merged or joined individual data sets correctly to create final stage data sets: Sample a few records, when individual data sets are available (most likely for intermediate data sets), from the individual data sets to confirm that they were joined properly. o Check for duplicate records and determine if record counts for the component data sets agree with those found in the merged data set. SOs calculated all data elements accurately: Recalculate the data fields that the SO used to calculate the data elements and refer to CMS guidance documents and TS. Calculate sums of the individual records within each reporting section to ensure that they equal those reported to CMS. Compare the data file created for submission to CMS with the data entered in the HPMS to confirm no manual data entry errors. For file uploads, confirm that the data file adheres to the record layout specified in HPMS. For the reporting sections that require reporting at the PBP level, if the DVC finds that the SO did not accurately enter and/or upload data reported for any of the PBPs included under a contract, then the reviewer must assign a “No” finding in the EES (Appendix B) for the entire contract for the applicable data element(s) for Sub-Standard 3.a and 3.b. If a reporting section requires both a file upload and data entry, both occur for an SO to meet Sub-Standard 3.a and 3.b. DVCs to confirm that the data does not have any logical errors. The DVC must perform data integrity checks to verify at the data element level. These data integrity checks include confirming that a data element does not include outlier records.

4. STANDARD 4-7

Standards 4 through 7 assess policies and procedures for periodic data system archiving and updates; an SO will most likely have these policies and procedures in place for an entire reporting section, as opposed to having them in place for only certain data elements. Exhibit 12 shows example rows from the EES (Appendix B) and guidance for Standards 4 through 7.

EXHIBIT 12. EXAMPLE ROWS FROM EES (APPENDIX B) FOR STANDARDS 4 THROUGH 7

Standard/ Sub- Standard ID	Standard/Sub-standard Description	Guidance
4	All source, intermediate, and final stage data sets and other outputs relied upon to enter data into CMS systems are archived.	• Determine if the SO has a policy or procedure for archiving all source, intermediate, and final stage data sets relied upon to enter data into CMS systems and confirm that the SO implemented this policy for the reporting section under review.
5	SO implements policies and procedures for the following: Periodic data system updates (for example, changes in enrollment, provider/pharmacy status, and claims adjustments). Restoring data in each data system (for example, disaster recovery plan).	Determine if the SO has policies and procedures in place for performing periodic updates for each data system used for the reporting section under review that ensures reported data are accurate and timely. Determine if the SO has policies and procedures in place for archiving and restoring data in each data system used for the reporting section under review that ensures timely data submission or re-submission in the event of data loss. Determine if the SO implements and adheres to the policies and procedures referenced above (such as, was any data for the reporting section under review negatively impacted by a failure to implement or follow these policies and procedures?).
6	If SO’s data systems underwent any changes during the reporting period (for example, because of a merger, acquisition, vendor change or upgrade): • SO provided documentation on the data system changes and, upon review, changes were implemented correctly and did not adversely impact the reported data.	• Review documentation on data system changes and determine if changes to an SO’s data system adversely impacted data reported by conducting the following activities: Determine if there were any changes to data sources used for data collection and storage, data processing, analysis, and reporting for the reporting section under review. Determine if data system changes were the root cause of any outlier notices received from CMS for the reporting section under review. Determine if the SO implemented any process or quality improvement activities during the reporting period specifically related to the data system change for the reporting section under review. Determine if any changes to data systems during the reporting period adversely impacted the validity of the SO’s data.

EXHIBIT 12. EXAMPLE ROWS FROM EES (APPENDIX B) FOR STANDARDS 4 THROUGH 7 (CONT.)

Standard/ Sub- Standard ID	Standard/Sub-standard Description	Guidance
7	If data collection and/or reporting for this reporting section is delegated to another entity: • SO regularly and routinely monitors the quality and timeliness of the data collected and/or reported by the delegated entity or first tier/downstream contractor.	Assess the following if data collection and/or reporting for a reporting section is delegated to another entity: Determine if the SO has policies and procedures in place for overseeing the delegated entity’s reporting process/results for the reporting section under review. o Determine if the SO implements and adheres to the policies and procedures referenced above (such as, was any data for the reporting section under review negatively impacted by a failure to implement or follow these policies and procedures?). SOs are not expected to replicate the delegated entity’s process and recalculate all their numbers but are expected to have policies and procedures in place for routine monitoring to ensure quality and timeliness of the data. CMS expects that these policies and procedures are implemented as frequently as needed to verify the delegated entities’ reporting. SOs are responsible for a delegated entities calculations and numbers and therefore if they’re incorrect, the responsibility ultimately falls on the SO.

3. Submit DV Review Findings via HPMS PRDVM

   1. Review Draft Findings with Sponsoring Organization

The SO and DVC should build time into the April-June 15 DV schedule to allow sufficient time for the review of the DV findings. Any issues identified during the review of the DVC findings must be resolved prior to the data validation contractor’s June 15 deadline for submitting findings to CMS. To maintain independence of the DV audit completed by the DVC, the review of draft findings between the SO and the DVC should be limited to technical clarifications and/or corrections required for factual accuracy of the DV review. SOs should not provide new data or request changes to the findings that are not evidence-based. The formal appeals process described in Section 6.2.3. remains an avenue for dispute resolution.

2. Submit DV Review Findings via HPMS PRDVM

Following the conclusion of the DV review and the finalization of findings, the DVC must report the findings by uploading the EES (Appendix B) to CMS via the PRDVM in HPMS by June 15th. The PRDVM Quick Reference Guide contains instructions for using this module, which is available in the PRDVM. The EES (Appendix B) includes review results and/or data sources that were reviewed for each standard or substandard, as well as the Yes, No, or Not Applicable finding, or a 1-5 Likert scale associated with each standard or substandard.

DVCs should also indicate which extraction method (full census or sample) was used for each standard.

6. POST-DV ACTIVITIES

   1. Compile Archive of DV Work Papers

The DVC must prepare a complete archive of work papers associated with the annual DV and provide it to the SO. At a minimum, this archive must contain the documentation described in Exhibit 13. The DVC should also retain a complete copy of this archive in accordance with its contract with the SO.

When the SO receives the archive from the DVC, the SO must add the documentation of its DVC selection process to the archive, including how its chosen DVC meets the minimum qualifications, credentials, and resources listed in Appendix A. Federal regulations require the SO to retain this complete archive for the 10-year retention period. CMS may ask for copies of Appendix I and work papers associated with the annual DV. CMS reviews the information to verify compliance with the data validation process pursuant to 42 CFR §423.514(j) and OMB control number 0938-1115.

EXHIBIT 13. MINIMUM DOCUMENTATION REQUIRED FOR DV ARCHIVE

	DV	ARCHIVE
• • • • • • •	Documentation of Data Validation Contractor Selection Process Documentation of completion of CMS Data Validation Training for all staff assigned to the data validation team Completed OAI (Appendix E), including all documentation provided in response to OAI Final Site Visit Agenda Completed Sign-in Sheets from site visit (if used) Final IDG used during site visit	Copies of any formal presentations during site visit Notes on staff interviews and demonstrations during site visit Census/sample data Additional documentation provided by SO during/after site visit Draft findings in EES (Appendix B) Notes on issues resulting in changes to draft findings Final EES (Appendix B)

2. Review Final DV Results

   1. Pass/Not Pass Determination

For each of the standards, sub-standards, and/or data elements, the DVC records the appropriate

“Yes/No” finding or a score using a 1-5 Likert scale. The EES findings are submitted to CMS via the PRDVM in HPMS. Then, CMS analyzes the findings submitted by the DVC and applies a scoring methodology to make Pass/Not Pass determinations.

CMS assigns percentage points for each finding and these can vary depending on the standard, substandard, and/or data element being scored in each section. A “No” or 1, 2, or 3 finding, however, will always result in a score of zero percentage points. The Data Validation Pass/Not Pass Determination Methodology (Appendix J) identifies the individual score CMS has assigned to each standard, substandard, and/or data element for all reporting sections. Additionally, after the DVC has submitted findings in HPMS, the SO can view the findings via the “Review Data Validation Findings Report”.

SOs can also view their final DV results in HPMS. To access this page, from the top menu select

“Monitoring,” then “Plan Reporting Data Validation.” Select the appropriate contract year. Select the

PRDVM Reports. Select “Score Detail Report.” Select the applicable reporting section. If you cannot see

the Plan Reporting Data Validation module, contact HPMS_Access@cms.hhs.gov.

2. Passing DV – Minimum threshold

CMS has established 95 percent as the passing DV threshold for all reporting sections. SOs may view their individual contract’s validation results in HPMS.

3. SO Appeal of DV Determination

The SO may submit an appeal if they disagree with the DVC’s findings. SOs must submit appeals 5 business days following the DV deadline.

An SO has the right to appeal:

• Reporting section score of less than 95 percent

• Non-compliant data validation standards/sub-standards, such as a "No" or a 1, 2, or 3 on the 5-point Likert scale, in the specific data element's data validation

If the SO wishes to appeal, it must submit an appeal 5 business days following the DV deadline to CMS. Submissions must be sent to CMS via the PartCandD_Data_Validation@cms.hhs.gov resource mailbox and must contain all the following information in order to be considered:

• Email subject line must state: “Data Validation: Appeal”

• Content of email must include the information below, in list format and in the following order:

  ▪	Name of SO
  ▪	CMS contract number(s)
  ▪	SO’s contact name, title, phone number and email address
  ▪	Name of DVC organization

• For each appeal, list the following information:

▪	Justification for appeal
▪	Include as attachment any documentation supporting the justification for appeal. The documentation must have been in existence at the time of the DV. For example, if after the DV, the SO resubmits corrected data, revises a policy and procedure, or corrects a programming code that caused it to improperly calculate reported data; the SO cannot submit documentation of these corrections to appeal.

Once CMS receives the appeal, we will carefully consider the justification and any supporting documentation. CMS has not established a timeframe for the consideration of SO appeals.

Glossary of Terms

Acronym	Description
CEO	Chief Executive Officer
CMS	Centers for Medicare and Medicaid Services
CY	Contract Year
DV	Data Validation
DVC	Data Validation Contractor
EES	Examination Engagement Standards
EUA	Enterprise User Administration
HIPAA	Health Insurance Portability and Accountability
HPMS	Health Plan Management System
IDG	Interview Discussion Guide
Manual	Procedure Manual
OAI	Organizational Assessment Instrument
PRDVM	Plan Reporting Data Validation Module
PRM	Plan Reporting Module
RR	Reporting Requirements
SO	Sponsoring Organization
TS	Technical Specifications

1 See 42 CFR §422.516(g) and §423.514(j)

2 If a PBP (Plan) under a contract terminates at any time in the CY reporting period and the contract remains active through July 1 of the following year, the contract must still report data for all PBPs, including the terminated PBP. If otherwise applicable, the contract must also undergo DV.

3 If a contract terminates before July 1 in the following year after the contract year (CY) reporting period, CMS does not require the contract to report any data for the respective two years – the CY reporting period, and the following year.

4 Standard 6 is not scored.