Anais do IHC'2001 - Departamento de Informática e Estatística - UFSC

74
Anais do IHC’2001 - IV Workshop sobre Fatores Humanos em Sistemas Computacionais
was divided into two phases. The first phase involved 27 non-expert people, without any
experience in software development, web design or usability evaluations, but familiar with
computers and web sites, while the second phase involved 13 software engineers and eight
usability experts.
Method
At first, the non-expert evaluators were assigned, at random, to five different groups: one
called Control group (CNTL), and four other groups, each corresponding to a different
inspection method. The Control group did not follow any method or document, while the
others received the following documents related to these inspection methods:
• ISO - ISO 9241-10 Dialogue principles (ISO, 1996);
• EC - Ergonomic Criteria (Bastien & Scapin, 1993);
• INDEX - Usability Index Checklist for Websites (Keevil, 1998);
• HEU - Corporate Portal Heuristics (Dias, 2001).
All evaluators were invited to read the method's document, and to identify usability
problems on the sample portal's interface, initially by exploring it on their own and then, by
trying to perform three typical tasks. The tasks provided to all evaluators had been
identified by the researcher, as being representative of typical tasks, based on detailed
analysis of the context of use of the sample portal. The participants were then given an
evaluation guide elaborated for this study, and asked to inspect, using the respective
methods' documents, all web pages they walked through to complete the typical tasks.
Finally, each evaluator reported the usability problems they encountered on a
common evaluation form, correlating each problem to a usability principle described in the
method's document they used. On the same form, the evaluators answered 17 demographic
questions, and expressed their opinion about the examined corporate portal and the
usability evaluation method used, answering 13 satisfaction statements (on a semantic
differential scale, with two opposite adjectives along each statement).
The usability problems identified through use of the four methods were categorized
using common metrics, so data could be compared across methods on dimensions like type
of usability problems (general or recurring problems; problems of low, moderate or high
severity; and problems associated to usability factors).
To compare the evaluation methods, the following criteria, gathered from previous
comparative studies, like Jeffries et al. (1991) and Bastien, Scapin & Leulier (1996), were
defined in advance: average time spent by evaluators; researcher time spent on evaluation
planning, sessions and data analysis; number and type of identified usability problems;
number of correct correlations between identified problems and method's principles; size of
aggregates (usability problems found by different number of evaluators); evaluators' doubts
and average opinion about the methods.
Based on the preliminary results of the first phase, in regard to the comparison
criteria above, two methods were selected to proceed on the second phase: the Corporate
Portal Heuristics (HEU), and the Usability Index Checklist for Websites (INDEX). The
software engineers were allocated to two groups, each corresponding to one of the
methods. The same occurred to the usability experts. All four groups were invited to try to