Behavioural Surveillance Surveys - The Wisdom of Whores

Calculating standard errors
with multi-stage cluster
designs
In order to test the statistical significance of
observed changes or trends, it is necessary to
have estimates of the magnitude of sampling
error associated with the survey estimates,
commonly referred to as standard errors.
The estimation of sampling depends upon
the sample design used in collecting the data.
As sample designs become more complex
(e.g., when stratification, cluster sampling, and
multiple stages of sample selection are used),
the procedures for estimating standard errors
become quite complicated. The estimation of
standard errors for such designs is beyond the
scope of this Guide, and consultation with a
statistician is recommended.
Unfortunately, standard statistical software
packages such as SPSS and EPI-INFO do not
provide an adequate solution to this problem.
While both packages will estimate the standard
errors of observed changes on indicators
and perform appropriate statistical tests, the
standard errors produced by these software
packages assume that simple random sampling
was used in gathering the survey data. Since
it is highly probable that cluster sampling will
be employed in BSS, the estimated standard
errors produced by these packages will be
incorrect. Because the standard errors
generated by standard software packages will
usually be under-estimated (because they
assume simple random sampling), there is the
danger that observed changes in indicators
will be believed to be statistically significant
when in fact they are not.
To avoid this problem, and perform careful
data analysis, software that can perform cluster
analysis should be used. STATA and SUDAAN
are two such packages. These programs do
not assume that simple random sampling
was used. They make use of the cluster
information to calculate design effects, and
adjust standard errors before conducting
statistical tests, thereby avoiding the problem
of incorrect conclusions.
If using appropriate software is not an
option, one other possibility is to compensate
for the expected under-estimation of standard
errors by tightening the criteria used for
judging statistical significance. For example,
instead of using p