Current Population Survey Design and Methodology - Census Bureau

Chapter 16.
Quality Indicators of Nonsampling Errors
(Updated coverage ratios, nonresponse rates, and other measures of quality can be found by clicking on ‘‘Quality Measures’’ at
.)
INTRODUCTION
Chapter 15 contains a description of the different sources
of nonsampling error in the CPS and the procedures
intended to limit those errors. In the present chapter, several
important indicators of potential nonsampling error
are described. Specifically, coverage ratios, response variance,
nonresponse rates, mode of interview, time-insample
biases, and proxy reporting rates are discussed. It
is important to emphasize that, unlike sampling error,
these indicators show only the presence of potential nonsampling
error, not an actual degree of nonsampling error
present.
Nonetheless, these indicators of nonsampling error are
regularly used to monitor and evaluate data quality. For
example, surveys with high nonresponse rates are judged
to be of low quality, but the actual nonsampling error of
concern is not the nonresponse rate itself, but rather nonresponse
bias, that is, how the respondents differ from the
nonrespondents on the variables of interest. Although it is
possible for a survey with a lower nonresponse rate to
have a larger nonresponse bias than a survey that has a
higher nonresponse rate (if the difference between respondents
and nonrespondents is larger in the survey with the
lower nonresponse rate than it is in the survey with the
higher nonresponse rate), one would generally expect that
larger nonresponse indicates a greater potential for bias.
While it is relatively easy to measure nonresponse rates, it
is extremely difficult to measure or even estimate nonresponse
bias. Thus, these indicators are simply a measurement
of the potential presence of nonsampling errors. We
are not able to quantify the effect the nonsampling error
has on the estimates, and we do not know the combined
effect of all sources of nonsampling error.
COVERAGE ERRORS
When conducting a sample survey, the primary goal is to
give every person in the target universe a known probability
of being selected for the sample. When this occurs, the
survey is said to have 100 percent coverage. This is rarely
the case, however. Errors can enter the system during
almost any phase of the survey process, from frame creation
to interviewing. A bias in the survey estimates
results when characteristics of people erroneously
included or excluded from the survey differ from those of
individuals correctly included in the survey. Historically in
the CPS, the net effect of coverage errors has been an
Current Population Survey TP66
U.S. Bureau of Labor Statistics and U.S. Census Bureau
underestimate of the size of the total population for most
major demographic population subgroups before the
population controls are applied (known as undercoverage).
Coverage Ratios
One way to estimate the coverage error present in a survey
is to compute a coverage ratio. A coverage ratio is the
outcome of dividing the estimated number of people in a
specific demographic group from the survey by an independent
population total for that group. The CPS coverage
ratios are computed by dividing a CPS estimate using the
weights after the first-stage ratio adjustment by the independent
population controls used to perform the national
and state coverage adjustments and the second-stage
ratio adjustment. See Chapter 10 for more information on
computation of weights. Population controls are not error
free. A number of approximations or assumptions are
required in deriving them. See Appendix C for details on
how the controls are computed. Chapter 15 highlighted
potential error sources in the population controls. Undercoverage
exists when the coverage ratio is less than 1.0
and overcoverage exists when the ratio is greater than
1.0. Figure 16−1 shows the average monthly coverage
ratios for September 2001 through September 2004.
In terms of race, Whites have the highest coverage ratio
(90.7 percent), while Blacks have the lowest (82.2 percent).
Females across all races have higher coverage ratios
than males. Hispanics 1 also have relatively low coverage
rates. Historically, Hispanics and Blacks have lower coverage
rates than Whites for each age group, particularly the
20−29 age group. This is by no fault of the interviewers or
the CPS process. These lower coverage rates for minorities
affect labor force estimates because people who are
missed by the CPS are on the average likely to be different
from those who are included. People who are missed are
accounted for in the CPS, but they are given the same
labor force characteristics as those of the people who are
included. This produces bias in the CPS estimates.
This graph, as well as two other graphs of coverage ratios
by race and gender, can be found at . (Their
updates, with more current data, will be posted on this
site as they are made available.) The three graphs provide
1 Hispanics may be any race.
Quality Indicators of Nonsampling Errors 16–1