Large-Scale Semi-Supervised Learning for Natural Language ...

amples from unlabeled data in order to train better models of selectional preference (Chapter6) and string similarity (Chapter 7). The discriminative classifiers trained from this dataexploited several novel sources of information, including character-level (string and capitalization)features for selectional preferences, and features derived from a character-basedsequence alignment for discriminative string similarity. While automatic example generationwas not applied to web-scale unlabeled data in this work, it promises to scale easilyto web-scale text. For example, after summarizing web-scale data with N-gram statistics,we can create examples using only several gigabytes of compressed, N-gram-text, ratherthan using the petabytes of raw web text directly. So automatic example generation fromaggregate statistics promises both better scaling and cleaner data (since aggregate statisticsnaturally exclude phenomena that occur purely due to chance).The key methods of parts one and two are, of course, compatible in another way: itwould be straightforward to use the output of the pseudo-trained models as features insupervised systems. This is similar to the approach of Ando and Zhang [2005], and, in fact,was pursued in some of our concurrent work [Bergsma et al., 2009a] (with good results).8.2 The Impact of this WorkWe hope the straightforward but effective techniques presented in this dissertation will helppromote simple, scalable semi-supervised learning as a future paradigm for NLP research.We advocate such a direction for several reasons.First, only via machine learning can we combine the millions of parameters that interactin natural language processing. Second, only by leveraging unlabeled data can we gobeyond the limited models that can be learned from small, hand-annotated training sets.Furthermore, it is highly advantageous to have an NLP system that both benefits fromunlabeled data and that can readily take advantage of even more unlabeled data when it becomesavailable. Both the volume of text on the web and the power of computer architecturecontinue to grow exponentially over time. Systems that use unlabeled data will thereforeimprove automatically over time, without any special annotation, research, or engineeringeffort. For example, in [Pitler et al., 2010], we presented a parser whose performance improveslogarithmically with the number of unique N-grams in a web-scale N-gram corpus.A useful direction for future work would be to identify other problems that can benefit fromthe use of web-scale volumes of unlabeled data. We could hopefully thereby enable an evengreater proportion of NLP systems to achieve automatic improvements in performance.The following section describes some specific directions for future work, and notessome tasks where web-scale data might be productively exploited.Once we find out, for a range of tasks, just how far we can get with big data and MLalone, we will have a better handle on what other sources of linguistic knowledge mightbe needed. For example, we can now get to around 75% accuracy on preposition selectionusing N-grams alone (Section 3.5). To correct preposition errors with even higher accuracy,we needed to exploit knowledge of the speaker’s native language (and thus their likelypreposition confusions), getting above 95% accuracy in this manner (but also sacrificing asmall but perhaps reasonable amount of coverage). It’s unlikely N-gram data alone wouldever allow us to select the correct preposition in phrases like, “I like to swim before/afterschool.” Similarly, we argued that to perform even better on non-referential pronoun detection(Section 3.7), we will need to pay attention to wider segments of discourse.109