2012 PROCEEDINGS - Public Relations Society of America

satisfaction matching the themes that surfaced in the initial focus groups. When the researcher
was confident in the quality of the questions, an entire draft of the instrument, including the
directions and the formatted questions, was completed. A small example of the evolution of the
questions, from initial theme suggestions to those that were used in the pilot tests, is shown in
Table 2. During the study, several pages of tables were constructed showing this evolution and
improving the instrument.
This preliminary instrument was introduced to two more focus groups of public relations
professionals. These participants were asked to utilize an instrument used by the American
Evaluation Association (AEA) in an effort to survey the clarity and quality of the constructed
instrument (AEA, 2010). Specifically, the AEA‘s Independent Consulting Topical Interest
Group uses the ―Peer Review Rubric‖ to ascertain input about newly created survey instruments
and to help them develop into reliable and valid instruments. Participants‘ input, along with
memos from the focus groups, was reviewed, and changes to the survey were made based on
participant feedback. This pilot process was repeated as necessary until a point of saturation was
reached.
To implement pilot tests, the researcher drafted a final survey and a former client of
COMM140, as well as clients from other firms/applied courses at other universities, were
observed while utilizing the instrument. This happened in individual instances. As the purpose
of pilot testing is to reveal if both the directions that the researcher provides and the questions
that the researcher asks are clear (Fowler, 2009), this last phase ensured that those that have been
in an actual client/student relationship understood the instrument, the questions, and had little
negative feedback about the instrument. If relevant negative feedback existed, the instrument
was revised as necessary and pilot tests were repeated, again, until a point of saturation was
reached. Again, the interaction with the former clients was audio recorded for more thorough
review, and the AEA instrument was utilized again.
Participants
The sample of this study for focus groups and pilot tests, respectively, was a purposive
sample of practitioners and former clients of COMM140 and other like clients at different
universities. Purposive sampling is constructed for a unique reason and is meant to serve a
specific purpose (Fink, 2009). It is ―a nonprobability sample in which individuals are
deliberately selected for inclusion based on their special knowledge, position, characteristics, or
relevant dimensions of the population‖ (p. 344).
The focus group participants for this study, totaling 44, were public relations practitioners
having experience ranging from 1.5 to 40 years, with over 360 total years of experience, with no
participants being repeated. The pilot test participants were clients of four different universitys‘
public relations program. Table 3 illustrates the participant breakdown.
Data Analysis Procedures
For this study, memos and recordings from focus groups were reviewed. Thematic
analysis was used while coding concepts that surfaced in the first focus groups. Specifically, this
analysis involved ―looking for similar word or statement clusters‖ (McDavid & Hawthorn, 2006,
p.174) that guided question construction and tool construction of this evaluative tool. The same
qualitative grouping-style analysis took place for all focus groups and for the pilot tests.
Results
A valid and reliable instrument was produced. The instrument, the Appendix, also
10