Large-Scale Semi-Supervised Learning for Natural Language ...

generated word clusters. Several researchers have used the hierarchical Brown et al. [1992]clustering algorithm, and then created features for cluster membership at different levels ofthe hierarchy [Miller et al., 2004; Koo et al., 2008]. Rather than clustering single words,Lin and Wu [2009] use phrasal clusters, and provide features for cluster membership whendifferent numbers of clusters are used in the clustering.Features for the Output of Auxiliary ClassifiersAnother way to create features from unlabeled data is to create features for the output ofpredictions on auxiliary problems that can be trained solely with unlabeled data [Ando andZhang, 2005]. For example, we could create a prediction for whether the word arena occursin a document. We can take all the documents where arena does and does not occur, andbuild a classifier using all the other words in the document. This classifier may predictthat arena does occur if the words hockey, curling, fans, etc. occur. When the predictionsare used as features, if they are useful, they will receive high weight at training time. Attest time, if we see a word like curling, for example, even though it was never seen in ourlabeled set, it may cause the predictor for arena to return a high score, and thus also causethe document to be recognized as sports.Note that since these examples can be created automatically, this problem (and otherauxiliary problems in the Ando and Zhang approach) fall into the category of those withNatural Automatic Examples as discussed above. One possible direction for future work isto construct auxiliary problems with pseudo-negative examples. For example, we couldinclude the predictions of various configurations of our selectional-preference classifier(Chapter 6) as a feature in a discriminatively-trained language model. We took a similarapproach in our work on gender [Bergsma et al., 2009a]. We trained a classifier onautomatically-created examples, but used the output of this classifier as another feature ina classifier trained on a small amount of supervised data. This resulted in a substantial gainin performance over using the original prediction on its own: 95.5% versus 92.6% (but noteother features were combined with the prediction of the auxiliary classifier).Features used in this DissertationIn this dissertation, we create features from unsupervised data in several chapters and inseveral different ways. In Chapter 6, to assess whether a noun is compatible with a verb,we create features for the noun’s distribution only with other verbs. Thus we characterizea noun by its verb contexts, rather than its full distribution, using less features than anaive representation using the noun’s full distributional profile. Chapters 3 and 5 also selectivelyuse features from parts of the total distribution of a word, phrase, or pair of words (tocharacterize the relation between words, for noun compound bracketing and verb tag disambiguationin Chapter 5). In Chapter 3, we characterize contexts by using selected typesfrom the distribution of other words that occur in the context. For the adjective-orderingwork in Chapter 5, we choose an order based on the distribution of the adjectives individuallyand combined in a phrase. Our approaches are simple, but effective. Perhaps mostimportantly, by leveraging the counts in a web-scale N-gram corpus, they scale to makeuse of all the text data on the web. On the other hand, scaling most other semi-supervisedtechniques to even moderately-large collections of unlabeled text remains “future work” fora large number of published approaches in the machine learning and NLP literature.34