Large-Scale Semi-Supervised Learning for Natural Language ...

More documents

Recommendations

Info

SystemPrecKlementiev-Roth (KR) L≤2 58.6KR L≤2 (normalized, boundary markers) 62.9phrases L≤2 61.0phrases L≤3 65.1phrases L≤3 + mismatches 65.6phrases L≤3 + mismatches + NED 65.8Table 7.2: Bitext French-English development set cognate identification 11-pt average precision(%).BitextDictionarySystem Fr Es De Fr Es De Gr Jp RsPREFIX 34.7 27.3 36.3 45.5 34.7 25.5 28.5 16.1 29.8DICE 33.7 28.2 33.5 44.3 33.7 21.3 30.6 20.1 33.6LCSR 34.0 28.7 28.5 48.3 36.5 18.4 30.2 24.2 36.6NED 36.5 31.9 32.3 50.1 40.3 23.3 33.9 28.2 41.4PREFIX+DICE+LCSR+NED 38.7 31.8 39.3 51.6 40.1 28.6 33.7 22.9 37.9[Kondrak, 2005]: LCSF 29.8 28.9 29.1 39.9 36.6 25.0 30.5 33.4 45.5[Ristad and Yianilos, 1998] 37.7 32.5 34.6 56.1 46.9 36.9 38.0 52.7 51.8[Tiedemann, 1999] 38.8 33.0 34.7 55.3 49.0 24.9 37.6 33.9 45.8[Klementiev and Roth, 2006] 61.1 55.5 53.2 73.4 62.3 48.3 51.4 62.0 64.4Alignm-Based Discrim. 66.5 63.2 64.1 77.7 72.1 65.6 65.7 82.0 76.9Table 7.3: Bitext, Dictionary Foreign-to-English cognate identification 11-pt average precision(%).from a dictionary for another project): 87.4% were above 0.58. We also checked cognationon 100 randomly-sampled, positively-labeled French-English pairs from both the dictionaryand bitext data (i.e. translated or aligned and having LCSR≥0.58). 100% of the dictionarypairs and 93% of the bitext pairs were cognate.Next, we investigate various configurations of the discriminative systems on one of ourcognate identification development sets (Table 7.2). The original [Klementiev and Roth,2006] (KR) system can be improved by normalizing the feature count by the longer stringlength and including the boundary markers. This is therefore done with all the alignmentbasedapproaches. Also, because of the way its features are constructed, the KR system islimited to a maximum substring length of two (L≤2). A maximum length of three (L≤3) inthe KR framework produces millions of features and prohibitive training times, while L≤3is computationally feasible in the phrasal case, and increases precision by 4.1% over thephrases L≤2 system. 4 Including mismatches results in another small boost in performance(0.5%), while using an edit distance feature again increases performance by a slight margin(0.2%). This ranking of configurations is consistent across all the bitext-based developmentsets; we therefore take the configuration of the highest scoring system as our Alignment-Based Discriminative system for the remainder of this paper.4 At the time of this research, preliminary experiments using even longer phrases (beyond L≤3) produceda computationally prohibitive number of features for SVM learning. Deploying feature selection techniquesmight enable the use of even more expressive and powerful feature sets with longer phrase lengths.103
0.711-pt Average Precision0.60.50.40.30.20.10NEDTiedemannRistad-YanilosKlementiev-RothAlignment-Based Discrim.1000 10000 100000 1e+06Number of training pairsFigure 7.2: Bitext French-English cognate identification learning curve.We next compare the Alignment-Based Discriminative scorer to the various other implementedapproaches across the three bitext and six dictionary-based cognate identificationtest sets (Table 7.3). The table highlights the top system among both the non-adaptive andadaptive similarity scorers. 5 In each language pair, the alignment-based discriminative approachoutperforms all other approaches, but the KR system also shows strong gains overnon-adaptive techniques and their re-weighted extensions. This is in contrast to previouscomparisons which have only demonstrated minor improvements with adaptive over traditionalsimilarity measures [Kondrak and Sherif, 2006].We consistently found that the original KR performance could be surpassed by a systemthat normalizes the KR feature count and adds boundary markers. Across all the test sets,this modification results in a 6% average gain in performance over baseline KR, but is stillon average 5% below the Alignment-Based Discriminative technique, with a statisticallysignificantly difference on each of the nine sets. 6Figure 7.2 shows the relationship between training data size and performance in ourbitext-based French-English data. Note again that the Tiedemann and Ristad & Yanilossystems only use the positive examples in the training data. Our alignment-based similarityfunction outperforms all the other systems across nearly the entire range of trainingdata. Note also that the discriminative learning curves show no signs of slowing down:performance grows logarithmically from 1K to 846K word pairs.For insight into the power of our discriminative approach, we provide some of ourclassifiers’ highest and lowest-weighted features (Table 7.4). Note the expected correspondencesbetween foreign spellings and English (k-c, f-ph), but also features that lever-5 Using the training data and the SVM to weight the components of the PREFIX+DICE+LCSR+NED scorerresulted in negligible improvements over the simple average on our development data.6 Following [Evert, 2004], significance was computed using Fisher’s exact test (at p = 0.05) to compare then-best word pairs from the scored test sets, where n was taken as the number of positive pairs in the set.104
Page 1 and 2:
University of AlbertaLarge-Scale Se
Page 5 and 6:
Table of Contents1 Introduction 11.
Page 7 and 8:
7 Alignment-Based Discriminative St
Page 9 and 10:
List of Figures2.1 The linear class
Page 11 and 12:
drawn in by establishing a partial
Page 13 and 14:
(2) “He saw the trophy won yester
Page 15 and 16:
actual sentence said, “My son’s
Page 17 and 18:
Uses Web-Scale N-grams Auto-Creates
Page 19 and 20:
spelling correction, and the identi
Page 21 and 22:
Chapter 2Supervised and Semi-Superv
Page 23 and 24:
emphasis on “deliverables and eva
Page 25 and 26:
Figure 2.1: The linear classifier h
Page 27 and 28:
The above experimental set-up is so
Page 29 and 30:
and discriminative models therefore
Page 31 and 32:
their slack value). In practice, I
Page 33 and 34:
One way to find a better solution i
Page 35 and 36:
Figure 2.2: Learning from labeled a
Page 37 and 38:
algorithm). Yarowsky used it for wo
Page 39 and 40:
Learning with Natural Automatic Exa
Page 41 and 42:
positive examples from any collecti
Page 43 and 44:
generated word clusters. Several re
Page 45 and 46:
One common disambiguation task is t
Page 47 and 48:
3.2.2 Web-Scale Statistics in NLPEx
Page 49 and 50:
For each target wordv 0 , there are
Page 51 and 52:
ut without counts for the class pri
Page 53 and 54:
Accuracy (%)10090807060SUPERLMSUMLM
Page 55 and 56:
We also follow Carlson et al. [2001
Page 57 and 58:
Set BASE [Golding and Roth, 1999] T
Page 59 and 60:
pronoun (#3) guarantees that at the
Page 61 and 62: 807876F-Score747270Stemmed patterns
Page 63 and 64: anaphoricity by [Denis and Baldridg
Page 65 and 66: ter, we present a simple technique
Page 67 and 68: We seek weights such that the class
Page 69 and 70: each optimum performance is at most
Page 71 and 72: We now show that ¯w T (diag(¯p)
Page 73 and 74: Training ExamplesSystem 10 100 1K 1
Page 75 and 76: Since we wanted the system to learn
Page 77 and 78: Chapter 5Creating Robust Supervised
Page 79 and 80: § In-Domain (IN) Out-of-Domain #1
Page 81 and 82: Adjective ordering is also needed i
Page 83 and 84: Accuracy (%)10095908580757065601001
Page 85 and 86: System IN O1 O2Baseline 66.9 44.6 6
Page 87 and 88: 90% of the time in Gutenberg. The L
Page 89 and 90: VBN/VBD distinction by providing re
Page 91 and 92: other tasks we only had a handful o
Page 93 and 94: without the need for manual annotat
Page 95 and 96: DSP uses these labels to identify o
Page 97 and 98: Semantic classesMotivated by previo
Page 99 and 100: empirical Pr(n|v) in Equation (6.2)
Page 101 and 102: Verb Plaus./Implaus. Resnik Dagan e
Page 103 and 104: SystemAccMost-Recent Noun 17.9%Maxi
Page 105 and 106: Chapter 7Alignment-Based Discrimina
Page 107 and 108: ious measures to learn the recurren
Page 109 and 110: how labeled word pairs can be colle
Page 111: Figure 7.1: LCSR histogram and poly
Page 115 and 116: Fr-En Bitext Es-En Bitext De-En Bit
Page 117 and 118: Chapter 8Conclusions and Future Wor
Page 119 and 120: 8.3 Future WorkThis section outline
Page 121 and 122: My focus is thus on enabling robust
Page 123 and 124: [Bergsma and Cherry, 2010] Shane Be
Page 125 and 126: [Church and Mercer, 1993] Kenneth W
Page 127 and 128: [Grefenstette, 1999] Gregory Grefen
Page 129 and 130: [Koehn, 2005] Philipp Koehn. Europa
Page 131 and 132: [Mihalcea and Moldovan, 1999] Rada
Page 133 and 134: [Ristad and Yianilos, 1998] Eric Sv
Page 135 and 136: [Wang et al., 2008] Qin Iris Wang,
Page 137: NNP noun, proper, singular Motown V
show all

Large-Scale Semi-Supervised Learning for Natural Language ...

Create successful ePaper yourself

Delete template?

Save as template?