Large-Scale Semi-Supervised Learning for Natural Language ...

automatic examples heuristically. We turn to this in the following subsection.Learning with Pseudo-Negative ExamplesWhile the previous section described problems where there were natural positive and negativeexamples (e.g., the correct accent marker is positive, while others, including no accent,are negative), there is a large class of problems in NLP where we only have positive examplesand thus it’s not clear how to use a discriminative classifier to evaluate new potentialexamples. This is the situation with seed data: you are presented with a list of only positiveseeds, and there’s nothing obvious to discriminate these from.In these situations, researchers have devised various ways to automatically create negativeexamples. For example, let us return to the example of hypernyms. Although Hearst [1992]started her algorithm with only a few examples, this was an unnecessary handicap. Thousandsof examples of hypernym pairs can be extracted automatically from the lexical databaseWordNet [Miller et al., 1990]. Furthermore, WordNet has good coverage of the relationsinvolving nouns that are actually in WordNet (as opposed to, obviously, no coverage ofrelations involving words that aren’t mentioned in WordNet at all). Thus, pairs of wordsin WordNet that are not linked in a hypernym structure can potentially be taken as reliableexamples of words that are not hypernyms (since both words are in WordNet, if theywere hypernyms, the relation would generally be labeled). These could form our negativeexamples for discrimination.Recognizing this, Snow et al. [2005] use WordNet to generate a huge set of both positiveand negative hypernym pairs: exactly what we need as training data for a large-scalediscriminative classifier. With this resource, we need not iteratively discover contexts thatare useful for hypernymy: Snow et al. simply include, as features in the classifier, all thesyntactic paths connecting the pair of words in a large parsed corpus. That is, they havefeatures for how often a pair of words occurs in constructions like,“Xs and other Ys, Yssuch as Xs, Ys including Xs, etc.” Discriminative training, not heuristic weighting, willdecide the importance of these patterns in hypernymy. To classify any new example pair(i.e., for nouns that are not in WordNet), we can simply construct their feature vector ofsyntactic paths and apply the classifier. Snow et al. [2005] achieve very good performanceusing this approach.This approach could scale to make use of features derived from web-scale data. For anypair of words, we can efficiently extract all the N-grams in which both words occur. Thisis exactly what we proposed for discriminating object and subject relations for Bears wonand trophy won in our example in Chapter 1, Section 1.3. We can create features from theseN-grams, and apply training and classification.We recently used a similar technique for classifying the natural gender of English nouns[Bergsma et al., 2009a]. Rather than using WordNet to label examples, however, we usedco-occurrence statistics in a large corpus to reliably identify the most likely gender of thousandsof noun phrases. We then used this list to automatically label examples in raw text,and then proceeded to learn from these automatically-labeled examples. This paper couldhave served as another chapter in this dissertation, but the dissertation already seemed sufficientlylong without it.Several other recent uses of this approach are also worth mentioning. Okanohara andTsujii [2007] created examples automatically in order to train a discriminative wholesentencelanguage model. Language models are designed to tell us whether a sequenceof words is valid language (or likely, fluent, good English). We can automatically gather31