Software Engineering for Students A Programming Approach

388 Chapter 31 ■ Assessing methods
In the experiment, 59 people were asked to test a 63-line PL/1 program. The people
were workers in the computer industry, most of whom were programmers, with an
average of 11 years experience in computing. They were told of a suspicion that the
program was not perfect and asked to test the program until they felt that they had
found all the errors (if any). An error meant a discrepancy between the program and
the specification. The people were provided with the program specification, the program
listing, a computer to run the program on and as much time as they wanted.
Different groups used different verification methods.
While the people were experienced and the program was quite small, their performance
was surprisingly bad. The mean number of bugs found was 5.7. The most errors
any individual found were 9. The least any person found was 3. The actual number of
bugs was 15. There were 4 bugs that no one found. The overwhelming conclusion
must be that people are not very effective at carrying out verification, whichever technique
they use.
Additional findings from this study were that the people were not careful enough in
comparing the actual output from the program with the expected outcome. Bugs that
were actually revealed were missed in this way. Also the people spent too long on testing
the normal conditions that the program had to encounter, rather than testing special
cases and invalid input situations.
The evidence from this and other experiments suggests that inspections are a very
effective way of finding errors. In fact inspections are at least as good a way of identifying
bugs as actually running the program (doing testing). So, if you had to choose
one method for verification, it would have to be inspection. Studies show that black box
testing and white box testing are roughly equally effective.
However, evidence suggests that the different verification techniques tend to discover
different errors. Therefore the more techniques that are employed the better – provided
that there is adequate time and money. So black box testing, white box testing and inspection
all have a role to play. If there is sufficient time and effort available, the best strategy
is to use all three methods.
The conclusion is that small-scale experiments can give useful insights into the effectiveness
of software development techniques.
Incidentally, there is another technique for carrying out verification, which has not
been assessed against the above techniques. Formal verification is very appealing because
of its potential for rigorously verifying a program’s correctness beyond all possible doubt.
However, it must be remembered that formal methods are carried out by fallible
human beings who make mistakes.
31.4 ● The current state of methods
The methods, tools and approaches discussed in this book are well-established and widely
used. There is a diversity of approaches to software development. This, of course,
reflects the infancy of the field. But it is also part of the joy of software engineering;
it would be boring if there was simply one process model, one design method, one