Large-Scale Semi-Supervised Learning for Natural Language ...

automatically-labeled examples are used to train the classifier. Unfortunately, automaticallylabeledexamples are often incorrect. The classifier works hard to classify these examplescorrectly, and subsequently gets similar examples wrong that it encounters at testing. If wehave enough manually-labeled examples, it seems that we want the ultimate mediator ofthe value of our features to be performance on these labeled examples, not performance onany pseudo-examples. This mediation is, of course, exactly what supervised learning does.If we instead create features from unlabeled data, rather than using unlabeled data to createnew examples, standard supervised learning can be used.How can we include information from unlabeled data as new features in a supervisedlearner? Section 2.2 described a typical feature representation: each feature is a binaryindicator of whether a word is present or not in a document to be classified. When we extractfeatures from unlabeled data, we add new dimensions to the feature representation. Thesenew dimensions are for features that represent what we might call second-order interactions– co-occurrences of words with each other in unlabeled text.In very recent papers, both Huang and Yates [2009] and Turian et al. [2010] providecomparisons of different ways to extract new features from unlabeled data; they both evaluateperformance on a range of tasks.Features Directly From a Word’s Distribution in Unlabeled TextReturning to our sports example, we could have a feature for whether a word in a givendocument occurs elsewhere, in unlabeled data, with the word score. A classifier could learnthat this feature is associated with the sports class, because words like hockey, baseball,inning, win, etc. tend to occur with score, and some of these likely occur in the training set.So, although we may never see the word curling during training, it does occur in unlabeledtext with many of the same words that occur with other sports terms, like the word score. Soa document that contains curling will have the second-order score feature, and thus curling,through features created from its distribution, is still an indicator of sports. Directly havinga feature for each item that co-occurs in a word’s distribution is perhaps the simplest way toleverage unlabeled data in the feature representation. Huang and Yates [2009] essentiallyuse this as their multinomial representation. They find it performs worse on sequencelabelingtasks than distributional representations based on HMMs and latent-semantic analysis(two other effective approaches for creating features from unlabeled data). One issuewith using the distribution directly is that although sparsity is potentially alleviated at theword level (we can handle words even if we haven’t seen them in training data), we increasesparsity at the feature level: there are more features to train but the same amount of trainingdata. This might explain why [Huang and Yates, 2009] see improved performance onrare words but similar performance overall. We return to this issue in Chapter 5 when wepresent a distributional representation for verb part-of-speech tag disambiguation that mayalso suffer from these drawbacks (Section 5.6).Features from Similar Words or Distributional ClustersThere are many other ways to create features from unlabeled data. One popular approachis to summarize the distribution of words (in unlabeled data) using similar words. [Wanget al., 2005] use similar words to help generalization in dependency parsing. [Marton etal., 2009] use similar phrases to help improve the handling of out-of-vocabulary terms in amachine translation system. Another recent trend is to create features from automatically-33