Current Population Survey Design and Methodology - Census Bureau

Hurwitz, and Bershad, 1961) and correlated response variance,
one form of which is interviewer variance (a measure
of the variability among responses obtained by different
interviewers over repeated administrations). Similarly,
when a particular design-estimator fails over repeated
sampling to include a particular set of population units in
the sampling frame or to ensure that all units provide the
required data, bias can be viewed as having components
such as coverage bias, unit nonresponse bias, or item nonresponse
bias (Groves, 1989). For example, a survey
administered solely by telephone could result in coverage
bias for estimates relating to the total population if the
nontelephone households were different from the telephone
households with respect to the characteristic being
measured (which almost always occurs).
One common theme of these types of models is the
decomposition of total mean squared error into two sets
of components, one resulting from the fact that estimates
are based on a sample of units rather than the entire
population (sampling error) and the other due to alternative
specifications of procedures for conducting the
sample survey (nonsampling error). (Since nonsampling
error is defined negatively, it ends up being a catch-all
term for all errors other than sampling error, and can
include issues such as individual behavior.) Conceptually,
nonsampling error in the context of statistical science has
both variance and bias components. However, when total
mean squared error is decomposed mathematically to
include a sampling error term and one or more other nonsampling
error terms, it is often difficult to categorize
such terms as either variance or bias. The term nonsampling
error is used rather loosely in the survey literature to
denote mean squared error, variance, or bias in the precise
mathematical sense and to imply error in the more general
sense of process mistakes (see next section).
Some nonsampling error components which are conceptually
known to exist have yet to be expressed in practical
mathematical models. Two examples are the bias associated
with the use of a particular set of interviewers and
the variance associated with the selection of one of the
numerous possible sets of questions. In addition, the estimation
of many nonsampling errors—and sampling
bias—is extremely expensive and difficult or even impossible
in practice. The estimation of bias, for example,
requires knowledge of the truth, which may be sometimes
verifiable from records (e.g., number of hours paid for by
employer) but often is not verifiable (e.g., number of
hours actually worked). As a consequence, survey organizations
typically concentrate on estimating the one component
of total mean squared error for which practical
methods have been developed—variance.
It is frequently possible to construct an unbiased estimator
of variance. In the case of complex surveys like the
CPS, estimators have been developed that typically rely on
the proposition— usually well-grounded—that the variability
among estimates based on various subsamples of the
one actual sample is a good proxy for the variability
among all the possible samples like the one at hand. In
the case of the CPS, 160 subsamples or replicates are used
in variance estimation for the 2000 design. (For more specifics,
see Chapter 14.) It is important to note that the estimates
of variance resulting from the use of this and similar
methods are not merely estimates of sampling
variance. The variance estimates include the effects of
some nonsampling errors, such as response variance and
intra-interviewer correlation. On the other hand, users
should be aware of the fact that for some statistics these
estimates of standard error might be statistically significant
underestimates of total error, an important consideration
when making inferences based on survey data.
To draw conclusions from survey data, samplers rely on
the theory of finite population sampling from a repeated
sampling perspective: If the specified sample designestimator
methodology were implemented repeatedly and
the sample size sufficiently large, the probability distribution
of the estimates would be very close to a normal distribution.
Thus, one could safely expect 90 percent of the estimates
to be within two standard errors of the mean of all possible
sample estimates (standard error is the square root
of the estimate of variance) (Gonzalez et al., 1975; Moore,
1997). However, one cannot claim that the probability is
.90 that the true population value falls in a particular interval.
In the case of a biased estimator due to nonresponse,
undercoverage, or other types of nonsampling error, confidence
intervals may not cover the population parameter at
the desired 90-percent rate. In such cases, a standard
error estimator may indirectly account for some elements
of nonsampling error in addition to sampling error and
lead to confidence intervals having greater than the nominal
90-percent coverage. On the other hand, if the bias is
substantial, confidence intervals can have less than the
desired coverage.
QUALITY MEASURES IN STATISTICAL PROCESS
MONITORING
The process of conducting a survey includes numerous
steps or components, such as defining concepts, translating
concepts into questions, selecting a sample of units
from what may be an imperfect list of population units,
hiring and training interviewers to ask people in the
sample unit the questions, coding responses into predefined
categories, and creating estimates that take into
account the fact that not everyone in the population of
interest had a chance to be in the sample and not all of
those in the sample elected to provide responses. It is a
process where the possibility exists at each step of making
a mistake in process specification and deviating during
implementation from the predefined specifications.
13–2 Overview of Data Quality Concepts Current Population Survey TP66
U.S. Bureau of Labor Statistics and U.S. Census Bureau