Large-Scale Semi-Supervised Learning for Natural Language ...

For each target wordv 0 , there are five 5-gram context patterns that may span it. For Example(1) in Section 3.1, we can extract the following 5-gram patterns:system tried to decidev 0tried to decidev 0 theto decidev 0 the twodecidev 0 the two confusablev 0 the two confusable wordsSimilarly, there are four 4-gram patterns, three 3-gram patterns and two 2-gram patternsspanning the target. With |F| fillers, there are 14|F| filled patterns with relevant N-gramcounts. For example, for F ={among,between}, there are two filled 5-gram patterns thatbegin with the word decide: “decide among the two confusable” and “decide between thetwo confusable.” We collect counts for each of these, along with all the other filled patternsfor this example. When F ={among,between}, there are 28 relevant counts for eachexample.We now describe various systems that use these counts.3.3.1 SUPERLMWe use supervised learning to map a target word and its context to an output. There aretwo steps in this mapping: a) converting the word and its context into a feature vector, andb) applying a classifier to determine the output class.In order to use the standard x,y notation for classifiers, we write things as follows:Let ¯x = Φ(V) be a mapping of the input to a feature representation, ¯x. We might alsothink of the feature function as being parameterized by the set of fillers, F and the N-gramcorpus, R, so that ¯x = Φ (F,R) (V). The feature function Φ (F,R) (·) outputs the count (inlogarithmic form) of the different context patterns with the different fillers. Each of thesehas a corresponding dimension in the feature representation. IfN = 14|F| counts are used,then each ¯x is anN-dimensional feature vector.Now, the classifier outputs the index of the highest-scoring candidate in the set of candidateoutputs, C = {c 1 ,c 2 ...,c K }. That is, we let y ∈ {1,...,K} be the set of classesthat can be produced by the classifier. The classifier, H, is therefore a K-class classifier,mapping an attribute vector, ¯x, to a class, y. Using the standard [Crammer and Singer,2001]-style multi-class formulation, H is parameterized by a K-by-N matrix of weights,W:KH W (¯x) = argmax{ ¯W r · ¯x} (3.1)r=1where ¯W r is the rth row of W. That is, the predicted class is the index of the row of Wthat has the highest inner-product with the attributes, ¯x. The weights are optimized using aset of M training examples, {(¯x 1 ,y 1 ),...,(¯x M ,y M )}.This differs a little from the linear classifier that we presented in Section 2.2. Here weactually have K linear classifiers. Although there is only one set of N features, there isa different linear combination for each row of W. Therefore, the weight on a particularcount depends on the class we are scoring (corresponding to the row of W, r), as well asthe filler, the context position, and the context size, all of which select one of the 14|F|base features. There are therefore a total of 14|F|K count-weight parameters. Chapter 4formally describes how these parameters are learned using a multi-class SVM. Chapter 4also discusses enhancements to this model that can enable better performance with fewertraining examples.40