Large-Scale Semi-Supervised Learning for Natural Language ...

Learning with Natural Automatic ExamplesSome of the lowest-hanging fruit in the history of NLP arose when researchers realized thatsome important problems in NLP could be solved by generating labeled training examplesautomatically from raw text.Consider the task of diacritic or accent restoration. In languages such as French orSpanish, accents are often omitted in informal correspondence, in all-capitalized text suchas headlines, and in lower-bit text encodings. Missing accents adversely affect both syntacticand semantic analysis. It would be nice to train a discriminative classifier to restorethese accents, but do we need someone to label the accents in unaccented text to provideus with labeled data? Yarowsky [1994] showed that we can simply take (readily-available)accented text, take the accents off and use them as labels, and then train predictors usingfeatures for everything except for the accents. We can essentially generate as many labeledexamples as we like this way. The true accent and the text provide the positive example.The unaccented or alternatively-accented text provides negative examples.We call these Natural Automatic Examples since they naturally provide the positive andnegative examples needed to solve the problem. We contrast these with problems in thefollowing section where, although one may have plentiful positive examples, one must usesome creativity to produce the negative examples.This approach also works for context-sensitive spelling correction. Here we try to determine,for example, whether someone who typed whether actually meant weather. Wetake well-edited text and, each time one of the words is used, we create a training example,with the word-actually-used as the label. We then see if we can predict these wordsfrom their confusable alternatives, using the surrounding context for features [Golding andRoth, 1999]. So the word-actually-used is the positive example (e.g. “whether or not”),while the alternative, unused words provide the negatives (e.g. “weather or not”). Bankoand Brill [2001] generate a lot of training data this way to produce their famous results onthe relative importance of the learning algorithm versus the amount of training data (theamount of training data is much much more important). In Chapter 3, we use this approachto generate data for both preposition selection and context-sensitive spelling correction.A similar approach could be used for training systems to segment text into paragraphs,to restore capitalization or punctuation, to do sentence-boundary detection (one must find anassiduous typist, like me, who consistently puts two spaces after periods, but only one afterabbreviations...), to convert curse word symbols like %*#@ back into the original curse,etc. (of course, some of these examples may benefit from a channel model rather thanexclusively a source/language model). The only limitation is the amount of training datayour algorithm can handle. In fact, by summarizing the training examples with N-grambasedfeatures as in Section 2.5.5 (rather than learning from each instance separately), therereally is no limitation on the amount of data you might learn from.There are a fairly limited number of problems in NLP where we can just create examplesautomatically this way. This is because in NLP, we are usually interested in generatingstructures over the data that are not surface apparent in naturally-occurring text. We returnto this when we discuss analysis and generation problems in Chapter 3. Natural automaticexamples abound in many other fields. You can build a discriminative classifier for whethera stock goes up or for whether someone defaults on their loan purely based on previousexamples. A search engine can easily predict whether someone will click on a search resultusing the history of clicks from other users for the same query [Joachims, 2002]. However,despite not having natural automatic examples for some problems, we can sometimes create30