Entropy Inference and the James-Stein Estimator, with Application to ...

More documents

Recommendations

Info

HAUSSER AND STRIMMER that p is fixed and known. In this setting, the Shannon entropy in natural units is given by 1 H = − p ∑ k=1 θ k log(θ k ). (1) In practice, the underlying probability mass function are unknown, hence H and θ k need to be estimated from observed cell counts y k ≥ 0. A particularly simple and widely used estimator of entropy is the maximum likelihood (ML) estimator Ĥ ML = − p ∑ k=1 constructed by plugging the ML frequency estimates ˆθ ML k log(ˆθ ML k ) ˆθ ML k = y k n (2) into Equation 1, with n = ∑ p k=1 y k being the total number of counts. In situations with n ≫ p, that is, when the dimension is low and when there are many observation, it is easy to infer entropy reliably, and it is well-known that in this case the ML estimator is optimal. However, in high-dimensional problems with n ≪ p it becomes extremely challenging to estimate the entropy. Specifically, in the “small n, large p” regime the ML estimator performs very poorly and severely underestimates the true entropy. While entropy estimation has a long history tracing back to more than 50 years ago, it is only recently that the specific issues arising in high-dimensional, undersampled data sets have attracted attention. This has lead to two recent innovations, namely the NSB algorithm (Nemenman et al., 2002) and the Chao-Shen estimator (Chao and Shen, 2003), both of which are now widely considered as benchmarks for the small-sample entropy estimation problem (Vu et al., 2007). Here, we introduce a novel and highly efficient small-sample entropy estimator based on James- Stein shrinkage (Gruber, 1998). Our method is fully analytic and hence computationally inexpensive. Moreover, our procedure simultaneously provides estimates of the entropy and of the cell frequencies suitable for plugging into the Shannon entropy formula (Equation 1). Thus, in comparison the estimator we propose is simpler, very efficient, and at the same time more versatile than currently available entropy estimators. 2. Conventional Methods for Estimating Entropy Entropy estimators can be divided into two groups: i) methods, that rely on estimates of cell frequencies, and ii) estimators, that directly infer entropy without estimating a compatible set of θ k . Most methods discussed below fall into the first group, except for the Miller-Madow and NSB approaches. 1. In this paper we use the following conventions: log denotes the natural logarithm (not base 2 or base 10), and we define 0log0 = 0. 1470
ENTROPY INFERENCE AND THE JAMES-STEIN ESTIMATOR 2.1 Maximum Likelihood Estimate The connection between observed counts y k and frequencies θ k is given by the multinomial distribution p n! Prob(y 1 ,...,y p ;θ 1 ,...,θ p ) = ∏ p k=1 y ∏ θ y k k k! . (3) Note that θ k > 0 because otherwise the distribution is singular. In contrast, there may be (and often are) zero counts y k . The ML estimator of θ k maximizes the right hand side of Equation 3 for fixed y k , leading to the observed frequencies ˆθ ML k = y k n with variances Var(ˆθ ML k ) = 1 n θ k(1 − θ k ) and Bias(ˆθ ML k ) = 0 as E(ˆθ ML k ) = θ k . 2.2 Miller-Madow Estimator While ˆθ ML k is unbiased, the corresponding plugin entropy estimator Ĥ ML is not. First order bias correction leads to Ĥ MM = Ĥ ML + m >0 − 1 , 2n where m >0 is the number of cells with y k > 0. This is known as the Miller-Madow estimator (Miller, 1955). 2.3 Bayesian Estimators Bayesian regularization of cell counts may lead to vast improvements over the ML estimator (Agresti and Hitchcock, 2005). Using the Dirichlet distribution with parameters a 1 ,a 2 ,...,a p as prior, the resulting posterior distribution is also Dirichlet with mean ˆθ Bayes k = y k + a k n+A , where A = ∑ p k=1 a k. The flattening constants a k play the role of pseudo-counts (compare with Equation 2), so that A may be interpreted as the a priori sample size. Some common choices for a k are listed in Table 1, along with references to the corresponding plugin entropy estimators, Ĥ Bayes = − p ∑ k=1 ˆθ Bayes k log(ˆθ Bayes k ). k=1 a k Cell frequency prior Entropy estimator 0 no prior maximum likelihood 1/2 Jeffreys prior (Jeffreys, 1946) Krichevsky and Trofimov (1981) 1 Bayes-Laplace uniform prior Holste et al. (1998) 1/p Perks prior (Perks, 1947) Schürmann and Grassberger (1996) √ n/p minimax prior (Trybula, 1958) Table 1: Common choices for the parameters of the Dirichlet prior in the Bayesian estimators of cell frequencies, and corresponding entropy estimators. 1471
Page 1: Journal of Machine Learning Researc
Page 5 and 6: ENTROPY INFERENCE AND THE JAMES-STE

Entropy Inference and the James-Stein Estimator, with Application to ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?