Current Population Survey Design and Methodology - Census Bureau

Item Response Analysis
The primary use of item response analysis was to determine
whether different questionnaires produce different
response patterns, which may, in turn, affect the labor
force estimates. Unedited data were used for this analysis.
Statistical tests were conducted to ascertain whether diferences
among the response patterns of different questionnaire
versions were statistically significant. The statistical
tests were adjusted to take into consideration the use of a
nonrandom clustered sample, repeated measures over
time, and multiple persons in a household.
Response distributions were analyzed for all items on the
questionnaires. The response distribution analysis indicated
the degree to which new measurement processes
produced different patterns of responses. Data gathered
using the other methods outlined above also aided interpretation
of the response differences observed. (Response
distributions were calculated on the basis of people who
responded to the item, excluding those whose response
was ‘‘don’t know’’ or ‘‘refused.’’)
Respondent Debriefings
At the end of the test interview, respondent debriefing
questions were administered to a sample of respondents
to measure respondent comprehension and response formulation.
From these data, indicators of how respondents
interpret and answer the questions and some measures of
response accuracy were obtained.
The debriefing questions were tailored to the respondent
and depended on the path the interview had taken. Two
forms of respondent debriefing questions were
administered— probing questions and vignette classification.
Question-specific probes were used to ascertain
whether certain words, phrases, or concepts were understood
by respondents in the manner intended (Esposito et
al., 1992). For example, those who did not indicate in the
main survey that they had done any work were asked the
direct probe ‘‘LAST WEEK did you do any work at all, even
for as little as 1 hour?’’ An example of the vignettes
respondents received is ‘‘Last week, Amy spent 20 hours
at home doing the accounting for her husband’s business.
She did not receive a paycheck.’’ Individuals were asked to
classify the person in the vignette as working or not working
based on the wording of the question they received in
the main survey (e.g., ‘‘Would you report her as working
last week not counting work around the house?’’ if the
respondent received the unrevised questionnaire, or
‘‘Would you report her as working for pay or profit last
week?’’ if the respondent received the current, revised
questionnaire (Martin and Polivka, 1995).
Behavior Coding
Behavior coding entails monitoring or audiotaping interviews
and recording significant interviewer and respondent
behaviors (e.g., minor/major changes in question
Current Population Survey TP66
U.S. Bureau of Labor Statistics and U.S. Census Bureau
wording, probing behavior, inadequate answers, requests
for clarification). During the early stages of testing, behavior
coding data were useful in identifying problems with
proposed questions. For example, if interviewers frequently
reword a question, this may indicate that the
question was too difficult to ask as worded; respondents’
requests for clarification may indicate that they were
experiencing comprehension difficulties; and interruptions
by respondents may indicate that a question was too
lengthy (Esposito et al., 1992).
During later stages of testing, the objective of behavior
coding was to determine whether the revised questionnaire
improved the quality of interviewer/respondent
interactions as measured by accurate reading of the questions
and adequate responses by respondents. Additionally,
results from behavior coding helped identify areas of
the questionnaire that would benefit from enhancements
to interviewer training.
Interviewer Debriefings
The primary objective of interviewer debriefing was to
identify areas of the revised questionnaire or interviewer
procedures that were problematic for interviewers or
respondents. The information collected was used to identify
questions that needed revision, and to modify initial
interviewer training and the interviewer manual. A secondary
objective was to obtain information about the questionnaire,
interviewer behavior, or respondent behavior
that would help explain differences observed in the labor
force estimates from the different measurement processes.
Two different techniques were used to debrief interviewers.
The first was the use of focus groups at the centralized
telephone interviewing facilities and in geographically
dispersed regional offices. The focus groups were
conducted after interviewers had at least 3 to 4 months
experience using the revised CPS instrument. Approximately
8 to 10 interviewers were selected for each focus
group. Interviewers were selected to represent different
levels of experience and ability.
The second technique was the use of a self-administered
standardized interviewer debriefing questionnaire. Once
problematic areas of the revised questionnaire were identified
through the focus groups, a standardized debriefing
questionnaire was developed and administered to all interviewers.
See Esposito and Hess (1992) for more information
on interviewer debriefing.
HIGHLIGHTS OF THE QUESTIONNAIRE REVISION
A copy of the questionnaire can be obtained from the
Internet at .
Design of the Current Population Survey Instrument 6–3