interpretation

e v a l u a t i n g s i g n e f f e c t i v e n e s s
analytic procedures, one researcher read through the interviews and examined them for
major ideas, patterns of responses, or themes. A final list of categories was developed into
which all responses were coded. Two people not involved with the study independently
coded a random sample of 10% of the interviews. The two independent coders both
coded only two statements (1%) differently than the first author. Eighty-seven percent
of 179 individual statements were coded identically by the three coders, and 12% of the
statements were agreed upon between two of the three coders, indicating high reliability.
Manipulation Checks
In order to be confident that each of the experimental signs was actually perceived by visitors
in the way it was intended by researchers, a series of manipulation checks was conducted.
Manipulation checks are an essential part of this type of evaluation since in order to assess
the effectiveness of a given communication product, researchers must be sure that the
product actually possesses the qualities it was designed to have. Speaking directly to the case
of research on vivid message design, O’Keefe (2003) makes a convincing argument that “in
order to be vivid, a message must be perceived as being vivid” (p. 243). And more specifically
in the present study, given that four different messages were theorized to be vivid or novel
in different ways, it was important to ascertain not only whether each of the messages was
perceived by visitors to be vivid, but whether it was perceived to have the qualities intended
by researchers to give it its vividness. In other words, was each of the messages reasonably
consistent with the theoretical basis behind its design?
To this end, a series of statements (e.g., “the text catches my attention,” “the
message makes me sympathize with bears”) was developed to determine if the variables
we were intending to manipulate with each individual treatment (e.g., emotion, novelty,
vividness) were perceived as such by respondents. Approximately 250 manipulation
checks (50 per treatment) were administered to randomly chosen campers and hikers
on days between observations and interviews. Respondents were asked to read the test
message and then respond to the statements using five 10-point agree/disagree scales
(1=strongly disagree, 10=strongly agree) and eight 10-point semantic differential items.
Results
Manipulation Checks
We anticipated the treatments would differ in the way they were perceived by
respondents, and significant differences were indeed found for 10 measures (Table 1).
Respondents agreed fairly strongly that all five treatments caught their attention. The
treatment designed to elicit sympathy for the bears (Attention Humans) did so more than
other signs; the treatment designed to create a mental image (My Bear Story) did so more
than other signs; the treatment designed to be humorous (Top 10 Reasons) was rated
as funnier than other signs; the treatment designed to have a telegraphic title (Leave it
in the Locker) was judged as having a more informative title than other signs; and the
standard park message was considered more typical than other signs. In sum, each of
the five messages was found to possess its intended distinguishing quality more so than
the same quality was possessed by the other messages. However, the four treatments
manipulated for vividness were not considered significantly more vivid than the control
(Park Message). It is possible that the physical characteristics (yellow sign) shared by all
five signs led to the similarity in “vividness” ratings.
v o l u m e 15, n u m b e r 1 25