data consistency, completeness and cleaning - The INCLEN Trust

Checking for Invalid Numeric Values• The techniques for checking invalid numeric data are quite differentfrom the techniques used with character data– Examine minimum and maximum values for each numeric variable– Internal consistency methods; if we see that most of the data values fallwithin a certain range of values, then any values that fall far enough outsidethe range may be data errors– Run a univariate analysis, focusing especially on• Number of non-missing observations, number of observation not equal to zero andthe number of observation greater than zero are of most interest at this stage• Extremes shows the five lowest and five highest values for numeric variables• Quantiles• Mean• Standard deviation to decide on constitute reasonable cutoffs for low and high datavalue• Range• Graphic displays: a stem-and leaf plot, a box plot and a normal probability plot• Check the medical records for the extreme values and write a note to thedata center about the findings to help in further cleaning of these data