Large-Scale Semi-Supervised Learning for Natural Language ...

More documents

Recommendations

Info

Table of Contents1 Introduction 11.1 What NLP Systems Do . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Writing Rules vs. Machine Learning . . . . . . . . . . . . . . . . . . . . . 21.3 Learning from Unlabeled Data . . . . . . . . . . . . . . . . . . . . . . . . 31.4 A Perspective on Statistical vs. Linguistic Approaches . . . . . . . . . . . 61.5 Overview of the Dissertation . . . . . . . . . . . . . . . . . . . . . . . . . 81.6 Summary of Main Contributions . . . . . . . . . . . . . . . . . . . . . . . 102 Supervised and Semi-Supervised Machine Learning in Natural Language Processing2.1 The Rise of Machine Learning in NLP . . . . . . . . . . . . . . . . . . . .12122.2 The Linear Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.3 Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.3.1 Experimental Set-up . . . . . . . . . . . . . . . . . . . . . . . . . 172.3.2 Evaluation Measures . . . . . . . . . . . . . . . . . . . . . . . . . 182.3.3 Supervised Learning Algorithms . . . . . . . . . . . . . . . . . . . 192.3.4 Support Vector Machines . . . . . . . . . . . . . . . . . . . . . . . 202.3.5 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.4 Unsupervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.5 Semi-Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 242.5.12.5.2Transductive Learning . . . . . . . . . . . . . . . . . . . . . . . .Self-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25272.5.3 Bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.5.4 Learning with Heuristically-Labeled Examples . . . . . . . . . . . 292.5.5 Creating Features from Unlabeled Data . . . . . . . . . . . . . . . 323 Learning with Web-Scale N-gram Models 353.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.2.1 Lexical Disambiguation . . . . . . . . . . . . . . . . . . . . . . . 363.2.2 Web-Scale Statistics in NLP . . . . . . . . . . . . . . . . . . . . . 383.3 Disambiguation with N-gram Counts . . . . . . . . . . . . . . . . . . . . .3.3.1 SUPERLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39403.3.2 SUMLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.3.3 TRIGRAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.3.4 RATIOLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.4 Evaluation Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.5 Preposition Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.5.1 The Task of Preposition Selection . . . . . . . . . . . . . . . . . . 433.5.2 Preposition Selection Results . . . . . . . . . . . . . . . . . . . . . 443.6 Context-Sensitive Spelling Correction . . . . . . . . . . . . . . . . . . . . 463.6.1 The Task of Context-Sensitive Spelling Correction . . . . . . . . . 463.73.6.2 Context-sensitive Spelling Correction Results . . . . . . . . . . . .Non-referential Pronoun Detection . . . . . . . . . . . . . . . . . . . . . .47483.7.1 The Task of Non-referential Pronoun Detection . . . . . . . . . . . 483.7.2 Our Approach to Non-referential Pronoun Detection . . . . . . . . 49
Page 1 and 2: University of AlbertaLarge-Scale Se
Page 6 and 7: 3.7.3 Non-referential Pronoun Detec
Page 8 and 9: List of Tables1.1 Summary of tasks
Page 10 and 11: Chapter 1IntroductionNatural langua
Page 12 and 13: These rules and scores might depend
Page 14 and 15: • won a trophy for:689• won the
Page 16 and 17: On the other hand, mapping language
Page 18 and 19: web-scale systems. This work was pu
Page 20 and 21: • since many more training exampl
Page 22 and 23: so within some branches of psycholo
Page 24 and 25: produces a feature vector. A vector
Page 26 and 27: problems. However, they do not work
Page 28 and 29: predicted classtrue class+1 -1+1 TP
Page 30 and 31: Having the largest possible margin
Page 32 and 33: and heuristic approaches are now be
Page 34 and 35: precise make-up and genre of the tr
Page 36 and 37: parsing and other tasks [Bergsma an
Page 38 and 39: which used some manual intervention
Page 40 and 41: automatic examples heuristically. W
Page 42 and 43: automatically-labeled examples are
Page 44 and 45: Chapter 3Learning with Web-Scale N-
Page 46 and 47: “Natural Automatic Examples,” a
Page 48 and 49: Franz, 2006]. This corpus simply li
Page 50 and 51: Here, we simply provide some intuit
Page 52 and 53: are available. This position-weight
Page 54 and 55:
MaxMin 2 3 4 52 50.2 63.8 70.4 72.6
Page 56 and 57:
Accuracy (%)100989694929088SUPERLMS
Page 58 and 59:
Pattern Filler TypeString#1: 3rd-pe
Page 60 and 61:
Accuracy (%)1009590858075706560SUPE
Page 62 and 63:
System P R F AccSUPERLM 80.0 73.3 7
Page 64 and 65:
Chapter 4Improved Natural LanguageL
Page 66 and 67:
y regularizing a classifier toward
Page 68 and 69:
Figure 4.1: Multi-class classificat
Page 70 and 71:
where ¯1 is an N-dimensional vecto
Page 72 and 73:
experiments, we took a very simple
Page 74 and 75:
Training ExamplesSystem 10 100 1KCS
Page 76 and 77:
ole a feature will play in predicti
Page 78 and 79:
1. Is there a benefit in combining
Page 80 and 81:
ourselves. 1 We chose Gutenberg and
Page 82 and 83:
System IN O1 O2[Malouf, 2000] 91.5
Page 84 and 85:
Accuracy (%)1009590858075706560N-GM
Page 86 and 87:
10095Accuracy (%)90858075701001e3N-
Page 88 and 89:
System IN O1 O2Baseline 70.5 66.8 8
Page 90 and 91:
System IN O1 O2Baseline 89.2 85.2 7
Page 92 and 93:
Chapter 6Discriminative Learning of
Page 94 and 95:
As described in Chapter 2, discrimi
Page 96 and 97:
court-martial are all mapped to the
Page 98 and 99:
SystemMacroAvg MicroAvg PairwiseP R
Page 100 and 101:
F-Score0.90.80.70.60.50.40.30.20.10
Page 102 and 103:
Interpolated Precision0.80.70.60.50
Page 104 and 105:
like the world wide web. We could a
Page 106 and 107:
word alignment [Kondrak et al., 200
Page 108 and 109:
Foreign Language F Words f ∈ F Co
Page 110 and 111:
aligned end characters (similar to
Page 112 and 113:
SystemPrecKlementiev-Roth (KR) L≤
Page 114 and 115:
Lang. Feat. Wt. ExampleFr (Bitext)
Page 116 and 117:
sets of cognates across groups of l
Page 118 and 119:
amples from unlabeled data in order
Page 120 and 121:
with standard lexical features coul
Page 122 and 123:
Bibliography[Adamson and Boreham, 1
Page 124 and 125:
[Brants and Franz, 2006] Thorsten B
Page 126 and 127:
[Daumé III, 2007] Hal Daumé III.
Page 128 and 129:
[Jiampojamarn et al., 2007] Sittich
Page 130 and 131:
[Litkowski and Hargraves, 2007] Ken
Page 132 and 133:
[Pantel and Lin, 2002] Patrick Pant
Page 134 and 135:
[Tetreault and Chodorow, 2008] Joel
Page 136 and 137:
Appendix APenn Treebank Tag SetTag
show all

Large-Scale Semi-Supervised Learning for Natural Language ...

Create successful ePaper yourself

Delete template?

Save as template?