Chapter 2. Minimum variance unbiased estimation

1.Minimum variance criterionAn unbiased estimator is defined by, with the importantrule that this holds for all possible values of the unknown parameter. Within this class ofestimators, the one with the minimum variance is the most searched.In searching for optimal estimators we need to adopt some optimality criterion. Anatural one is the mean square error (MSE):This measures the average mean squared deviation of the estimator from the truevalue.Adoption of this natural criterion leads to unrealizable estimators, ones thatcannot be written solely as a function of the data. To understand the problem which ariseswe first rewrite the MSE as:That shows that the MSE is composed of errors due to the variance of the estimator as wellas the bias.Generally, any criterion which depends on the bias will lead to an unrealizableestimator. Although this is generally true, on ocassion realizable minimum MSE estimatorscan be found.From a practical viewpoint the minimum MSE estimator needs to be abandoned.An alternative approach is to constrain the bias to be zero and find the estimator whichminimizes the variance. Such an estimator is called the minimum variance unbiased (MVU)estimator.Note: The MSE of an unbiased estimator is just the variance.Minimizing the variance of an unbiased estimator also has the effect ofconcentrating the PDF (probability density function) of the estimator error, , aboutzero. The estimation error will therefore be less likely to be large.3

3.Finding the minimum variance unbiased estimatorEven if a MVU estimator exists, we may not be able to find it. There is no known“turn-the crunk” procedure which will produce the estimator. There are, instead, a variety oftechniques that can sometimes be applied to find the MVUE.These methods include:1. Compute the Cramer-Rao Lower Bound (CRLB), and check the condition for equality.2. Apply the Rao-Blackwell-Lehmann-Scheffe (RBLS) theorem.3.Restrict the class of estimators to be not only unbiased but also linear. Then find theminimum variance estimator within this restricted class.Approaches 1. and 2. may produce the MVU estimator, while 3. will yield it onlyif the MVU estimator is linear in the data.The CRLB allows us to determine that for any unbiased estimator the variancemust be greater than or equal to a given value, as shown in Figure 3.Figure 3. Cramer-Rao lower bound on variance of unbiased estimatorIf an estimator exists whose variance equals the CRLB for each value of ө, thenit must be the MVU estimator. In this case the theory of the CRLB immediately yields theestimator. It may happen that no estimator exists whose variance equals the bound. Yet aMVU estimator may still exist, as for instance in the case of ө 1 in Figure 3. Then we mustresort to the Rao-Blackwell-Lehmann-Scheffen theorem. This procedure first find asufficient statistic, one which uses all the data efficiently, and then finds a function of asufficien statistic which is an unbiased estimator ө. With a slight restriction of the PDF ofthe data this procedure will then be garanteed to produce the MVU estimator. The thirdapproach requires the estimator to be linear, a sometimes severe restriction, and chooses thebest linear estimator. Of course, only for particular data sets can this approach produce theMVU estimator.6

4.Extension to a vector parameterIfis a vector of unknown parameters, then say that an estimatoris unbiased if:(*)for i= 1,2,3…,p. By definingwe can equivalently define an unbiased estimator to have the propertyfor every ө contained within the space defined in (*). A MVU estimator has the additionalproperty thatis minimum among all unbiased estimators.7

5.ConclusionsIn statistics, the mean square error or MSE of an estimator is one of many waysto quantify the difference between an estimator and the true value of the quantity beingestimated. MSE is a risk function, corresponding to the expected value of the squared errorloss or quadratic loss. MSE measures the average of the square of the "error." The error isthe amount by which the estimator differs from the quantity to be estimated. Note that theMSE is not equivalent to the expected value of the absolute error.Since MSE is an expectation, it is a scalar, and not a random variable. It may be afunction of the unknown parameter θ, but it does not depend on any random quantities.However, when MSE is computed for a particular estimator of θ the true value of which isnot known, it will be subject to estimation error.MVUE is an unbiased estimator that has lower variance than any other unbiasedestimator for all possible values of the parameter. An efficient estimator doesn’t need toexist, but if it does, it’s the MVUE. Since the mean square error (MSE) of an estimator isMSE(θ) = var(θ) + bias(θ)2the MVUE minimizes MSE among unbiased estimators .For an unbiased estimator, the MSE is the variance. Like the variance, MSE hasthe same unit of measurement as the square of the quantity being estimated. In some casesbiased estimators have lower MSE because they have a smaller variance than does anyunbiased estimator.The question of determining the MVUE, if one exists, for a particular problem isimportant for practical statistics, since less-than-optimal procedures would naturally beavoided, other things being equal. This has led to substantial development of statisticaltheory related to the problem of optimal estimation.8

6.References1. Fundamentals of Statistical Signal Processing : Estimation Theory, Steven M.Key,University of Rhode Island.2. Prediction and Improved Estimation in Linear Models, J. Wiley, New York, 1977.3. Unbiased estimators and Their Applications, V.G Voinov, M.S. Nikulin, KluwerAcademic Publishers9