Split Panel Test Plan -- ACS Internet Version

The purpose of this letter is to inform you of our plans to conduct additional research
under the generic clearance for questionnaire pre-testing research (OMB number 06070725). The Statistical Research Division will be conducting split panel testing to examine
whether the creation of a user account, consisting of a user ID and password, should be
required for participants to access the Internet version of the American Community
Survey (ACS) that is currently being developed for field testing around the year 2010.
This user ID and password would be an additional step to the login process, since
respondents will already be required to enter their ACS/household ID and PIN from the
paper form they receive in the mail.
Having the respondents create a user account would conform to Federal Desktop Core
Configuration Standards and would increase the security of their responses in the event
that they did not complete the survey on the first pass and needed to return later to finish
it. Specifically, it will be less likely that another user will be able to either accidentally or
intentionally access another respondent’s saved responses. Finishing the survey in more
than one visit to the Web site is a likely scenario for many respondents because the ACS
is a lengthy survey – the 2009 paper ACS survey indicates that the estimated time burden
for completion is 38 minutes, including time for reviewing the instructions and answers.
This investigation will focus on the login process in which respondents must engage
before being able to respond to the survey questions and complete the survey. The
testing has one primary goal: to determine whether there will be a difference in the user
experience (e.g., the time it takes for the respondent to reach the first question, user
satisfaction score, and rate of completion of login process) when given the option to
create a user account, consisting of a user ID and password, before taking the survey
versus when required to create such an account. In the optional account creation
condition, participants will be given the option to create an account again after they
attempt to leave the survey without completing it. We will also be collecting eyetracking data to examine which visual aspects of the login process were noticed and used
by the respondents.
Between March and April 2009, staff from the Census Bureau’s usability lab will conduct
a maximum of 40 usability interviews, 20 using the option treatment and 20 using the
required treatment. We will use novice participants and target the following participant
characteristics:
ƒ At least one year of Internet experience, with little or no exposure to the Census
Web site or to Census terminology;
ƒ Familiar and comfortable with searching for information on the Internet;
ƒ No known disabilities;
ƒ Any amount of education; and
ƒ Not a federal employee.
We will recruit from our internal database made up of respondents in the Metropolitan
DC area who have volunteered to participate in a usability study at the Census Bureau.
Participants will be randomly assigned to one of two conditions (i.e., optional account
creation [consisting of a user ID and password] vs. required account creation.)

The sessions will take place in the Census Bureau's usability lab. Each experimental
session will include a test administrator and one participant. Participants will be asked to
fill out an initial questionnaire with demographic questions on their age, sex, and
education level. The questionnaire also contains questions about the respondents’
computer and Internet experience and is included as Attachment A.
The testing procedure will consist of asking the participant to use a real ACS paper
version of the survey (including the mail envelope and label needed to start the login
process) to access the survey. A sample ACS envelope and label are included as
Attachment B, and a copy of the paper survey is included as Attachment C. Then,
depending on the condition, the participant will either be allowed the option of creating
an account with a user ID and password in order to access the ACS online form or will be
required to do so. Then, the participant will complete a small subset of questions using a
prototypical online ACS form following the login segment. Attachment D contains the
test protocol for this study, and Attachment E contains screen shots of the current
appearance of each step of the login process and a sample survey question. Following the
testing procedure, each participant will complete a Questionnaire for User Interface
Satisfaction (QUIS, Attachment F). We will read a set of debriefing questions aloud to
the participant. These questions can be found in Attachment G. Since the participant will
only be completing a few survey questions when they will be led to believe they will be
completing the whole survey, a debriefing statement will be read to the participants. This
statement is included as Attachment H.
Participants will be informed that their response is voluntary and that the information
they provide is confidential and will be seen only by employees involved in the research
project. Participants will be paid $40 for their assistance in this project.
The estimated time for completion of the experimental session is one hour. Thus, the
total estimated burden time for this research is a maximum of 40 hours.
The contact person for questions regarding data collection and statistical aspects of the
design of this research is listed below:
Kathleen T. Ashenfelter, Ph.D.
Center for Survey Methods Research
Room 5K020K
U.S. Census Bureau
Washington, D.C. 20233
(301) 763- 4922
Kathleen.T.Ashenfelter@Census.Gov