Large-Scale Semi-Supervised Learning for Natural Language ...

Here, we simply provide some intuitions on what kinds of weights will be learned. Tobe clear, note that ¯Wr , the rth row of the weight-matrix W, corresponds to the weights forpredicting candidate c r . Recall that in generation tasks, the set C and the set F may beidentical. So some of the weights in ¯W r will therefore correspond to features for patternsfilled with filler f r . Intuitively, these weights will be positive. That is, we will predictthe class among when there are high counts for the patterns filled with the filler among(c r =f r =among). On the other hand, we will choose not to pick among if the counts onpatterns filled with between are high. These tendencies are all learned by the learningalgorithm. The learning algorithm can also place higher absolute weights on the morepredictive context positions and sizes. For example, for many tasks, the patterns that beginwith a filler are more predictive than patterns that end with a filler. The learning algorithmattends to these differences in predictive power as it maximizes prediction accuracy on thetraining data.We now note some special features used by our classifier. If a pattern spans outside thecurrent sentence (when v 0 is close to the start or end), we use zero for the correspondingfeature value, but fire an indicator feature to flag that the pattern crosses a boundary. Thisfeature provides a kind of smoothing. Other features are possible: for generation tasks,we could also include synonyms of the output candidates as fillers. Features could also becreated for counts of patterns processed in some way (e.g. converting one or more contexttokens to wildcards, POS-tags, lower-case, etc.), provided the same processing can be doneto the N-gram corpus (we do such processing for the non-referential pronoun detectionfeatures described in Section 3.7).We call this approach SUPERLM because it is SUPERvised, and because, like an interpolatedlanguage model (LM), it mixes N-gram statistics of different orders to produce anoverall score for each filled context sequence. SUPERLM’s features differ from previouslexical disambiguation feature sets. In previous systems, attribute-value features flag thepresence or absence of a particular word, part-of-speech, or N-gram in the vicinity of thetarget [Roth, 1998]. Hundreds of thousands of features are used, and pruning and scalingcan be key issues [Carlson et al., 2001]. Performance scales logarithmically with thenumber of examples, even up to one billion training examples [Banko and Brill, 2001]. Incontrast, SUPERLM’s features are all aggregate counts of events in an external (web) corpus,not specific attributes of the current example. It has only 14|F|K parameters, for theweights assigned to the different counts. Much less training data is needed to achieve peakperformance. Chapter 5 contrasts the performance of classifiers with N-gram features andtraditional features on a range of tasks.3.3.2 SUMLMWe create an unsupervised version of SUPERLM. We produce a score for each filler bysumming the (unweighted) log-counts of all context patterns filled with that filler. Forexample, the score for among could be the sum of all 14 context patterns filled with among.For generation tasks, the filler with the highest score is taken as the label. For analysis tasks,we compare the scores of different fillers to arrive at a decision; Section 3.7.2 explains howthis is done for non-referential pronoun detection.We refer to this approach in our experiments as SUMLM.For generation problems where F =C, SUMLM is similar to a naive bayes classifier,41