Anais do IHC'2001 - Departamento de Informática e Estatística - UFSC

Anais do IHC’2001 - IV Workshop sobre Fatores Humanos em Sistemas Computacionais 81
Evaluators' average opinion about the methods
After the evaluation process itself, the participants of this study were asked to give their
subjective opinion about the methods they used. Comparing their answers to questions
associated to a semantic differential scale, it was verified that the Corporate Portal
Heuristics presented more positive and less negative aspects than the other methods, while
the Ergonomic Criteria got just the opposite.
The EC method received the most negative evaluation, being considered boring,
difficult, confusing and excessively detailed. The only question that this method got
exclusively positive ratings was about its text, considered well written by all evaluators.
One of the evaluators mentioned that this method should be less detailed and less
redundant in order to be well understood by readers.
The evaluators that used the Corporate Portal Heuristics, on the other hand,
considered the following positive aspects of this method : utility, effectiveness, context
compatibility, well written text, easiness, efficiency and flexibility. One evaluator
mentioned that this method is simple and easy to understand.
Based on the opinion of ISO group members, this method was referred as efficient,
effective, and with a well written text. However, some answers tended to negative
adjectives, like rigid, horrible and frustrating. One of the evaluators of this group
mentioned that this method was so complicated that he/she preferred to take part in the
control group and evaluated the sample portal following his/her own judgements.
The positive aspects of the Usability Index Checklist for Websites were : well
written text, efficiency, utility, effectiveness and context compatibility. However, around
40% of the evaluators considered it boring and excessively detailed. One Index evaluator
remarked that many items of this checklist were too general for this specific usability
inspection. Another evaluator commented that this method is a little bit confusing and
inflexible.
Conclusion
Considering the results of the comparative analysis, two evaluation methods were
remarkable in this case study: the Heuristic Evaluation (whose support tool was the
document entitled Corporate Portal Heuristics), and the Usability Index Checklist for
Websites, mainly for being able to identify more problems, and to cover all usability
factors and severities, when applied as complementary methods. The INDEX groups
identified more general, 'Error prevention', 'Flexibility and efficiency of use', 'Visibility,
recognition and conduction' problems, while the HEU groups were the best in identifying
severe, 'User control' and 'Aesthetics and simplicity' problems.
It was noticed that usability experts, using the Corporate Portal Heuristics, and
software engineers, using the Usability Index Checklist for Websites, were the evaluation
groups that identified more usability problems, and in less time, when compared to nonexpert
groups. The time spent by the researcher during the evaluation sessions and data
analysis was also smaller with usability experts and software engineers.
Taking into account the quality of each method’s document, and the evaluators’
subjective opinion about them, the Corporate Portal Heuristics got more positive results
than the other tested methods, and the usability expert group was the best one in