CASINO manual - Theory of Condensed Matter

B
Appendix 2: Automatic testing of CASINO
B.1 Overview
A facility for automatically testing casino can be found in CASINO/examples/TEST. This directory
contains another directory called Input, which holds a large number of examples designed to test most
aspects of casino, and a script called autotest, which will run casino for each of those examples
in turn. This facility is intended to help developers ensure that they do not inadvertently break the
code when adding new features.
B.2 Running the set of examples
The first time that autotest is run, a stable, trusted version of casino should be used to create a
library of standard output files, with which subsequent results can be compared. Type ./autotest
--library to achieve this. (Note that the casino binary compiled with debugging flags will be used
by default.)
Subsequently, just type ./autotest to run the examples and compare the output with that in the library.
The script will warn you if casino fails or halts unexpectedly, or if the output differs from that in
the library (don’t worry if it scrolls off the screen too fast; all output is saved in the autotest.log
file). Note that rounding errors often affect the last couple of digits in the results of optimization
calculations. Moreover, there are often trivial differences between the results produced with different
compilers, so, when testing the code, it is best to use the same compiler as was used to generate the
library.
The autotest script uses runqmc to run casino, and one may pass options to runqmc directly using
the syntax ‘-- 〈extra-runqmc-options〉 (i.e., use a double hyphen to separate autotest options
from the ones to be passed to runqmc, as in ‘runqmc --nproc=100’. This allows, for example, running
Shm and OpenMP autotests, controlling the number of cores used on of parallel autotests, running the
debug, opt, or dev versions of the code, and so on.
To skip a particular example use, e.g., ‘--not carbon atom’. To run the test for a particular example
only, use e.g. ‘--only neon atom’. Use --missing to only run tests which have not been run already,
or were interrupted; use --error to only run tests which have not been run already, or were interrupted,
or resulted in an error; use --mismatch to only run tests which have not been run already,
or were interrupted, or did not match the library output. If you want to get rid of all the results
(including the library results), either remove the Library and Output directories by hand, or type
./autotest --clean. Type ./autotest --help to get a full list of options.
The facility is good at picking up minor bugs when used in conjunction with the NAG compiler with
debugging flags. The facility should be run after any substantial modification of casino.
B.3 Adding a new example
If a new feature is added to casino then an appropriate example should be added to the automatic
testing facility. Create a new directory in Input with name describing the system, and create one
or more subdirectories in it with names indicative of the run type; see other examples for guidance.
Let’s refer to these directories as system and system/run, respectively. Place common files (e.g.,
xwfn.data files, pseudopotentials, etc) under system, and files which change from run to run (e.g.,
input) in the individual system/run subdirectories.
Note that the examples should run very quickly; it doesn’t matter whether good results will be
produced. The great majority of bugs will show up when the results of the modified version of
casino are compared with the ‘standard’ version.
B.4 Using git-bisect with autotest
Bisecting is a very simple, yet powerful, feature of git which aids locating the commit at which a
regression was introduced, and thus (hopefully) determine how to fix it.
Suppose you have a clean working directory (no modifications over the last commit), and when you
run the autotest you find a crash, an output mismatch, a large increase in CPU time or any other
217