Anais do IHC'2001 - Departamento de Informática e Estatística - UFSC

80
Anais do IHC’2001 - IV Workshop sobre Fatores Humanos em Sistemas Computacionais
reaching, after the third evaluator, 47 problems (39,2% of the total problems detected). To
reach this level, it was necessary to assign seven software engineers. The Non_E_Heu
curve implies that, even adding more evaluators, the number of problems detected by the
UE_Heu group would not be equalled.
Comparing the UE curves until the third evaluator, it was verified that both groups
reached the same number of problems at this point. The software engineers, on the other
hand, found a larger number of usability problems when using the Index method.
Average severity of usability problems identified by each method
Table 4 shows the average severity, in a scale from 0 to 3, of the usability problems found
by each group. There was a convergence to moderate severity. The Index groups, however,
reached a lower average severity, around 1,4. This finding is in accordance with Jeffries et
al. (1991), whose study points, as a disadvantage of inspections based on guidelines (or
checklists), the fact that they miss many high severity problems, when compared to
heuristic evaluation, for example. Among the Index groups, the software engineers found a
larger number of high severity problems (Table 5).
Non_E
_ISO
Non_E
_Cntl
Table 4 – Average severity.
Non_E
_EC
Non_E
_Heu
SE_
Heu
UE_
Heu
Non_E
_Index
SE_Index UE_Index
Average severity 1,8 1,72 1,75 1,74 1,6 1,72 1,4 1,4 1,37
Total of 18 high
severity problems
Number of high
severity problems
Table 5 – Identification of high severity problems.
Non_E
_ISO
Non_E
_Cntl
Non_E
_EC
Non_E
_Heu
SE_
Heu
UE_
Heu
Non_E
_Index
SE_Index UE_Index
4 6 4 9 7 10 5 7 6
Considering the 18 high severity problems (Table 5), rated by all evaluators, the
UE_Heu group was the one that detected the largest number, surpassing the Index group of
software engineers, that found the largest number of problems in total (Table 2).
Comparing only non-expert evaluators, the Heuristics group was also the one that
reached the largest number of problems in this range (50% of the high severity problems
detected by all groups), while the ISO and EC groups identified only four. One evaluator
of the Non_E_Heu group found 5 severe problems, a performance comparable or superior
to the total number of severe problems detected by Non_E_ISO, EC and Index groups (4, 4
and 5, respectively).
Taking into account the behavior of the aggregates’ curves for high severity
problems, this study also observed that there was a certain resemblance between groups
using the same method in relation to slope and stability points, similarly to what happened
with the total of usability problems, shown in Figure 1.