Small Area Estimation of Poverty

5. How reliable are these maps?5.1 tests for over-fittingTo ascertain that the coefficients estimated with the betaand alpha models capture the general relationshipsbetween expenditure and observables, rather than beingonly relevant for a limited set of survey households,all regions are subject to over-fitting tests. They arecarried out in accordance with their description in themethodology section. Thus, estimating the modelson a 50% subsample of the region, following the ELLmethodology, and comparing the predicted results inthe other 50% subsample with observed expenditure inthat sub-sample. Meanwhile, variables that were foundinsignificant on the first subsample were excluded fromthe models. Table 4 shows the results of this exercise. Thepredicted and observed FGT(0) is presented by the threeregions for which we have separate models, as well as bystrata within those regions. Z-values for all 22 categoriesare below the absolute value of two, suggesting thatour models are not over-fitting. Note that for the subregionalcategories, these Z-values must be taken with agrain of salt as their low number of observations (abouthalf of their normal stratum size, or less) may cause theconfidence interval to be relatively wide. But regardless,just looking at the total of the three regions gives usconfidence in the estimated results with Z-values of0.002, 1.016 and 0.220 for respectively Central&Eastern,Western, and Midwestern & Farwestern.Table 4: Over-fitting test comparing observed and predicted FGT(0) in a subsample of the NLSS3 surveyObserved FGT(0) in subsample 2 (survey Predicted FGT(0) in subsample 2Strata: N Mean S.E. mean Mean S.E. mean Z-valueCentral & Eastern region:Mountain 127 0.205 0.036 0.152 0.016 -1.330Urban-Kathma 422 0.104 0.015 0.134 0.008 1.744Urban-Hill 73 0.082 0.032 0.109 0.018 0.736Rural-Hill-Eastern 192 0.130 0.024 0.180 0.015 1.727Rural-Hill-Central 239 0.188 0.025 0.154 0.014 -1.175Urban-Terai 199 0.156 0.026 0.158 0.015 0.081Rural-Terai-Eastern 233 0.180 0.025 0.164 0.014 -0.564Rural-Terai-Central 237 0.215 0.027 0.191 0.014 -0.810Total Central & Eastern region 1722 0.157 0.009 0.157 0.005 0.002Western region:Urban-Hill 116 0.009 0.009 0.020 0.005 1.139Rural-Hill-Western 246 0.211 0.026 0.233 0.016 0.689Urban-Terai 45 0.089 0.043 0.079 0.026 -0.207Rural-Terai-Western 169 0.166 0.029 0.191 0.018 0.742Total Western region 576 0.148 0.015 0.166 0.010 1.016Midwestern & Farwestern region:Mountain 82 0.451 0.055 0.505 0.029 0.856Urban-Hill 37 0.189 0.065 0.252 0.039 0.820Rural-Hill-Midwestern 181 0.293 0.034 0.325 0.018 0.839Rural-Hill-Farwestern 87 0.414 0.053 0.354 0.027 -1.000Urban-Terai 88 0.295 0.049 0.270 0.029 -0.450Rural-Terai-Midwestern 125 0.192 0.035 0.222 0.022 0.712Rural-Terai-Farwestern 78 0.372 0.055 0.298 0.029 -1.181Urban-Terai 88 0.295 0.049 0.270 0.029 -0.450Rural-Terai-Midwestern 125 0.192 0.035 0.222 0.022 0.712Rural-Terai-Farwestern 78 0.372 0.055 0.298 0.029 -1.181Total Midwestern & Farwestern region 678 0.313 0.018 0.317 0.010 0.220Presenting observed and predicted poverty in a random subsample of the survey by stratum, where the predicted poverty rate is based ona consumption model calibrated on the other half of the survey data (“subsample 1”). Z-value = ( FGT(0) census - FGT(0) survey ) / √[ (S.E.census)2 + (S.E. census)2]. The value of Z should thus not exceed │2│for both measures to represent the same poverty incidence. Source: NLSSIII.NEPAL Small Area Estimation of Poverty, 2011, Summary and Major Findings 13