Attachment O -- Non-Response Bias Analysis of the 2014 MEPS-IC

Attachment O -- Non-response Bias Analysis of Private Establishments from the 2014 Medical
Expenditures Panel Survey – Insurance Component (MEPS-IC)
Introduction:
When an expected unit response rate is below 80 percent, OMB Standards & Guidelines for Statistical
Surveys recommends conducting a nonresponse bias analysis. Of the 42,055 sample units selected for
the 2014 MEPS-IC, 27,226 (64.7%) responded, 11,776 (28.0%) did not respond, and 3,053 (7.3%) were
out of sample or out of business. Removing the out of sample and out of business units from the
response rate calculation results in an unweighted response rate of 69.8 percent. As shown in the
formula below, nonresponse bias is a function of both the nonresponse rate and the difference between
the respondent mean and the nonrespondent mean on the variable of interest:

OR
Respondent Mean = Full Sample Mean + (Nonresponse Rate)*(Respondent Mean –Nonrespondent
Mean)
In the MEPS-IC we are most concerned about nonresponse bias in our key estimates- the percent of
establishments offering health insurance, the percent of employees offered health insurance and the
percent of employees enrolled in health insurance, among other important estimates. Unfortunately,
since we do not have these estimates for the nonresponding establishments, we cannot directly
measure the potential nonresponse bias in these estimates. However, from the sampling frame we
have data for both responding and nonresponding establishments that are correlated with, or vary by,
many of our key estimates. These variables include the size of the firm the establishment is in (number
of employees), the industry group the establishment belongs to and the region of the country where the
establishment is located (Census division). This analysis will compare the responding establishments to
the nonresponding establishments on these sampling frame variables, using both a chi-square test of
independence and a t-test to test differences in means and percentages.
The rest of this memo includes three sections where the differences between responding and
nonresponding establishments will be tested and discussed, followed by a discussion of the weighting
adjustments for nonresponse bias and a conclusion section.
Firm Size:
Firm size is highly correlated with at least one of our key measures, the percentage of establishments
that offer health insurance. In 2014, 25.7 percent of private sector establishments in firms with less
than 10 employees offered health insurance and this percentage increased to establishments in firms
with 1,000 or more employees where 99.2 percent offered insurance. Table 1 presents the results of a
chi-square test of the relationship between firm size and response. The test shows that response to the
MEPS-IC is not independent of firm size and this may be a source of nonresponse bias.
To identify which firms size categories are possibly the source of this bias, table 2 shows the percent
distribution of responding and nonresponding establishments across the firm size categories and the

results of testing the difference in these percentages. The results show those establishments in firms
with less than 10 employees, and those with 25 to 99 employees may be a source of nonresponse bias.
Industry Group:
Most of the MEPS-IC key estimates vary by industry group. For example, in 2014 the percent of
establishments that offered health insurance to their employees ranged from 23.6 percent for
establishments in agriculture, fishing and forestry to 61.8 percent for those in mining and
manufacturing. Table 3 presents the results of a chi-square test of the relationship between industry
group and response. The test shows that response to the MEPS-IC is not independent of industry
category and this may be a source of nonresponse bias.
To identify which industry category is possibly the source of this bias, table 4 shows the percent
distribution of responding and nonresponding establishments across the industry categories and the
results of testing the difference in these percentages. The results show those establishments in
agriculture, fishing and forestry, mining and manufacturing, construction, and professional services may
be a source of nonresponse bias.
Census Division:
Many of the MEPS-IC key estimates vary by Census division. For example, in 2013 the percent of
employees in establishments that offered health insurance ranged from 82.0 percent for establishments
located in West South Central to 88.6 percent for those located in New England. Table 5 presents the
results of a chi-square test of the relationship between Census region and response. The test shows
that response to the MEPS-IC is not independent of Census division and this may be a source of
nonresponse bias.
To identify which Census division is possibly the source of this bias, table 6 shows the percent
distribution of responding and nonresponding establishments across divisions and the results of testing
the difference in these percentages. The results show those establishments located in 6 of the 9
divisions may be a source of nonresponse bias.
Weighting adjustments for nonresponse bias:
The base sampling weights of the respondents to the MEPS-IC are adjusted so that the respondents also
represent the nonrespondents while minimizing the bias associated with nonresponse. The adjustment
is made by controlling firm size, establishment size, industry group, type of firm, and state. Thus, a
nonresponding establishment is represented by a responding establishment with characteristics similar
to the extent possible in terms of these variables. A raking procedure is applied to adjust the weights of
the respondents to represent all eligible establishments on the frame (i.e., both respondents and
nonrespondents) while controlling for the marginal distributions of all these variables. The raking
adjustment is expected to reduce any bias due to nonresponse to the extent the MEPS-IC estimates are
associated with the characteristics used in the raking procedure. Since the MEPS-IC estimates are
generally highly correlated with these characteristics, the weighting adjustment is expected to minimize
the nonresponse bias to a large extent.

Conclusion:
The results of this analysis show that there is the potential for nonresponse bias in the MEPS-IC.
Although we never really know the extent of any bias in the survey estimates, since the distributions of
responding and nonresponding establishments are close, and since the weighting adjustment takes into
account the important variables by which MEPS-IC estimates mostly vary, we can be fairly confident
that, to the extent possible, nonresponse bias has been addressed in the MEPS-IC.

Frequency
Expected
Percent
Row Pct
Col Pct

Table 1. Chi-Square of Response by Firm Size, 2014 MEPS-IC
Firm Size
Responding (N)
Nonresponding (N)
Total
Less than 10
2653849
1263299 3917148
2685550
1231598
39.96
19.02
58.98
67.75
32.25
58.29
60.5
10 to 24
550138
240744 790882
542219
248663
8.28
3.63
11.91
69.56
30.44
12.08
11.53
25 to 99
377294
146313 523607
358979
164628
5.68
2.2
7.88
72.06
27.94
8.29
7.01
100 to 999
298031
138730 436762
299439
137323
4.49
2.09
6.58
68.24
31.76
6.55
6.64
1,000 or more
673818
298987 972805
666944
305861
10.15
4.5
14.65
69.27
30.73
14.8
14.32
Total
4553130
2088073 6641204
68.56
31.44
100
Statistic
DF
Value
Prob
Chi-Square
4
4776.6482
<.0001
Cramer's V
0.0268

Table 2. T-test of Response by Firm Size, 2014 MEPS-IC
Firm Size
Less than 10
10 to 24
25 to 99
100 to 999
1,000 or more
Total

Frequency
Expected
Percent
Row Pct
Col Pct

Responding (%)
58.31
12.1
8.28
6.54
14.77
100

Nonresponding (%)
60.47
11.48
7.01
6.65
14.39
100

DF
22377
39000
23839
27276
22425

t
Value
-4
1.76
4.4
-0.4
0.98

Pr >
|t|
<.0001
0.0776
<.0001
0.6883
0.328

Table 3. Chi-Square of Response by Industry, 2014 MEPS-IC
Industry
Responding (N) Nonresponding (N)
Total
Agriculture,
118102
63347.5 181450
Fishing, and Forestry
124400
57050
1.78
0.95
2.73
65.09
34.91
2.59
3.03
Mining and
206529
83937.3 290466
Manufacturing
199140
91326
3.11
1.26
4.37
71.1
28.9
4.54
4.02
Construction
362858
201415 564273
386859
177414
5.46
3.03
8.5
64.31
35.69
7.97
9.65
Utilities and
137421
61984 199405
Transportation
136710
62695
2.07
0.93
3
68.92
31.08
3.02
2.97
Wholesale Trade
241713
114359 356072
244119
111953
3.64
1.72
5.36
67.88
32.12
5.31
5.48
Financial Services
528676
237594 766270
and Real Estate
525346
240924
7.96
3.58
11.54
68.99
31.01
11.61
11.38

Retail Trade

643184
637053
9.68
69.22
14.13
1178993
1156460
17.75
69.89
25.89
1135655
1143044
17.1
68.12
24.94
4553130
68.56

Professional Services

Other

Total
Statistic
Chi-Square
Cramer's V

DF

Value
8

286023 929207
292154
4.31
13.99
30.78
13.7
507821 1686814
530354
7.65
25.4
30.11
24.32
531592 1667246
524202
8
25.1
31.88
25.46
2088073 6641204
31.44
100
Prob
8512.2193 <.0001
0.0358

Table 4. T-test of Response by Industry, 2014 MEPS-IC
Industry
Agriculture, Fishing, and Forestry
Mining and Manufacturing
Construction
Utilities and Transportation
Wholesale Trade
Financial Real Estate
Retail Trade
Professional Services
Other
Total

Responding (%)
2.59
4.54
7.99
3.01
5.31
11.59
14.11
25.92
24.94
100

Nonresponding (%)
3.04
4.01
9.61
2.98
5.48
11.42
13.74
24.25
25.47
100

DF
20723
23455
20628
22297
21899
22334
22434
39000
22070

t
Value
-2.44
2.4
-5.11
0.15
-0.7
0.48
0.98
3.51
-1.07

Pr >
|t|
0.0146
0.0165
<.0001
0.8811
0.4823
0.6341
0.3284
0.0005
0.2863

Frequency
Expected
Percent
Row Pct
Col Pct

Table 5. Chi-Square of Response by Division, 2014 MEPS-IC
Division
Responding (Y) Nonresponding (N)
Total
New
240544
102128 342672
England
234932
107740
3.62
1.54
5.16
70.2
29.8
5.28
4.89
Middle
587908
335436 923344
Atlantic
633034
290310
8.85
5.05
13.9
63.67
36.33
12.91
16.06
East
670924
289825 960749
North
658678
302071
Central
10.1
4.36
14.47
69.83
30.17
14.74
13.88
West
388453
137034 525487
North
360268
165219
Central
5.85
2.06
7.91
73.92
26.08
8.53
6.56
South
861378
416445 1277823
Atlantic
876060
401763
12.97
6.27
19.24
67.41
32.59
18.92
19.94
East
242474
92219.5 334693
South
229462
105231
Central
3.65
1.39
5.04
72.45
27.55
5.33
4.42
West
480372
227581 707952
South
485364
222589
Central
7.23
3.43
10.66
67.85
32.15
10.55
10.9
Mountain
355633
140220 495853
339951
155902
5.35
2.11
7.47
71.72
28.28
7.81
6.72

Pacific

725446
735383
10.92
67.63
15.93
4553130
68.56

Total
Statistic
ChiSquare
Cramer's
V

DF

Value
8

347185 1072630
337248
5.23
16.15
32.37
16.63
2088073 6641204
31.44
100
Prob
24415.1746

<.0001

0.0606

Table 6. T-test of Response by Geographic Division, 2014 MEPS-IC
Division
Responding (%)
Nonresponding (%)
DF
t Value
New England
5.3
4.85 39000
1.84
Middle Atlantic
12.92
16.07 20505
-7.97
East North Central
14.74
13.87 39000
2.27
West North Central
8.52
6.59 24796
6.77
South Atlantic
18.92
19.93 21814
-2.3
East South Central
5.33
4.41 24142
3.94
West South Central
10.54
10.92 21901
-1.09
Mountain
7.8
6.74 23623
3.72
Pacific
15.93
16.62 21868
-1.68
Total
100
100

Pr > |t|
0.066
<.0001
0.0231
<.0001
0.0216
<.0001
0.2777
0.0002
0.0939