The HTK Book Steve Young Gunnar Evermann Dan Kershaw ...

14.2 Statistically-derived Class Maps 195predicted. Considering for clarity the bigram 6 case, then given G(.) the language model has theterms w i , w i−1 , G(w i ) and G(w i−1 ) available to it. The probability estimate can be decomposedas follows:P class’ (w i | w i−1 ) = P (w i | G(w i ), G(w i−1 ), w i−1 )× P (G(w i ) | G(w i−1 ), w i−1 ) (14.7)It is assumed that P (w i | G(w i ), G(w i−1 ), w i−1 ) is independent of G(w i−1 ) and w i−1 and thatP (G(w i ) | G(w i−1 ), w i−1 ) is independent of w i−1 , resulting in the model:P class (w i | w i−1 ) = P (w i | G(w i )) × P (G(w i ) | G(w i−1 )) (14.8)Almost all reported class n-gram work using statistically-found classes is based on clusteringalgorithms which optimise G(.) on the basis of bigram training set likelihood, even if the class mapis to be used with longer-context models. It is interesting to note that this approximation appearsto works well, however, suggesting that the class maps found are in some respects “general” andcapture some features of natural language which apply irrespective of the context length used whenfinding these features.14.2 Statistically-derived Class MapsAn obvious question that arises is how to compute or otherwise obtain a class map for use in alanguage model. This section discusses one strategy which has successfully been used.Methods of statistical class map construction seek to maximise the likelihood of the trainingtext given the class model by making iterative controlled changes to an initial class map – in orderto make this problem more computationally feasible they typically use a deterministic map.14.2.1 Word exchange algorithm[Kneser and Ney 1993] 7 describes an algorithm to build a class map by starting from some initialguess at a solution and then iteratively searching for changes to improve the existing class map.This is repeated until some minimum change threshold has been reached or a chosen number ofiterations have been performed. The initial guess at a class map is typically chosen by a simplemethod such as randomly distributing words amongst classes or placing all words in the first classexcept for the most frequent words which are put into singleton classes. Potential moves are thenevaluated and those which increase the likelihood of the training text most are applied to the classmap. The algorithm is described in detail below, and is implemented in the HTK tool Cluster.Let W be the training text list of words (w 1 , w 2 , w 3 , . . .) and let W be the set of all words in W.From equation 14.1 it follows that:P class (W) =∏P class (x | y) C(x,y) (14.9)x,y∈Wwhere (x, y) is some word pair ‘x’ preceded by ‘y’ and C(x, y) is the number of times that the wordpair ‘y x’ occurs in the list W.In general evaluating equation 14.9 will lead to problematically small values, so logarithms canbe used:log P class (W) =∑C(x, y). log P class (x | y) (14.10)x,y∈WGiven the definition of a class n-gram model in equation 14.8, the maximum likelihood bigramprobability estimate of a word is:P class (w i | w i−1 ) =C(w i )C(G(w i )) × C (G(w i), G(w i−1 ))C(G(w i−1 ))(14.11)6 By convention unigram refers to a 1-gram, bigram indicates a 2-gram and trigram is a 3-gram. There is nostandard term for a 4-gram.7 R. Kneser and H. Ney, “Improved Clustering Techniques for Class-Based Statistical Language Modelling”;Proceedings of the European Conference on Speech Communication and Technology 1993, pp. 973-976