Large-Scale Semi-Supervised Learning for Natural Language ...

System IN O1 O2[Malouf, 2000] 91.5 65.6 71.6webc(a 1 ,a 2 ) vs. c(a 2 ,a 1 ) 87.1 83.7 86.0SVM with N-GM features 90.0 85.8 88.5SVM with LEX features 93.0 70.0 73.9SVM with N-GM + LEX 93.7 83.6 85.4Table 5.3: Adjective ordering accuracy (%). SVM and [Malouf, 2000] trained on BNC,tested on BNC (IN), Gutenberg (O1), and Medline (O2).Finally, we also have features for all suffixes of length 1-to-4 letters, as these encodeuseful information about adjective class [Malouf, 2000]. Like the adjective features, thesuffix features receive a value of +1 for adjectives in the first position and −1 for those inthe second.N-GM features[Lapata and Keller, 2005] propose a web-based approach to adjective ordering: take themost-frequent order of the words on the web, c(a 1 ,a 2 ) vs. c(a 2 ,a 1 ). We adopt this as ourunsupervised approach. We merge the counts for the adjectives occurring contiguously andseparated by a comma.These are the most useful N-GM features; we include them but also other, tag-basedcounts from Google V2. Raw counts include cases where one of the adjectives is notused as a modifier: “the special present was” vs. “the present special issue.” We includelog-counts for the following, more-targeted patterns: 5 c(a 1 a 2 N.*), c(a 2 a 1 N.*),c(DT a 1 a 2 N.*), c(DT a 2 a 1 N.*). We also include features for the log-counts of eachadjective preceded or followed by a word matching an adjective-tag: c(a 1 J.*), c(J.* a 1 ),c(a 2 J.*), c(J.* a 2 ). These assess the positional preferences of each adjective. Finally, weinclude the log-frequency of each adjective. The more frequent adjective occurs first in57% of pairs.As in all tasks, the counts are features in a classifier, so the importance of the differentpatterns is weighted discriminatively during training.5.3.2 Adjective Ordering ResultsIn-domain, with both feature classes, we set a strong new standard on this data: 93.7%accuracy for the N-GM+LEX system (Table 5.3). We trained and tested [Malouf, 2000]’sprogram on our data; our LEX classifier, which also uses no auxiliary corpus, makes 18%fewer errors than Malouf’s system. Our web-based N-GM model is also superior to thedirect evidence web-based approach of [Lapata and Keller, 2005], scoring 90.0% vs. 87.1%accuracy. These results show the benefit of both our new lexicalized and our new web-basedfeatures.Figure 5.1 gives the in-domain learning curve. With fewer training examples, the systemswith N-GM features strongly outperform the LEX-only system. Note that with tens ofthousands of test examples, all differences are highly significant.Out-of-domain, LEX’s accuracy drops a shocking 23% on Gutenberg and 19% on Medline(Table 5.3). [Malouf, 2000]’s system fares even worse. The overlap between training5 In this notation, capital letters (and reg-exps) are matched against tags while a 1 and a 2 match words.73