Large-Scale Semi-Supervised Learning for Natural Language ...

We also follow Carlson et al. [2001] and Chodorow et al. [2007] in extracting a subset ofdecisions where our system has higher confidence. We only propose a label if the ratio betweenthe highest and second-highest score from our classifier is above a certain threshold,and then vary this threshold to produce accuracy at different coverage levels (Figure 3.2).The SUPERLM system can obtain close to 90% accuracy when deciding on 70% of examples,and above 95% accuracy when deciding on half the examples. The TRIGRAMperformance rises more slowly as coverage drops, reaching 80% accuracy when decidingon only 57% of examples.Many of SUPERLM’s errors involve choosing between prepositions that are unlikely tobe confused in practice, e.g. with/without. Chodorow et al. [2007] wrote post-processorrules to prohibit corrections in the case of antonyms. Note that the errors made by an Englishlearner also depend on their native language. A French speaker looking to translateau-dessus de has one option in some dictionaries: above. A German speaker looking totranslate über has, along with above, many more options. When making corrections, wecould combine SUPERLM (a source model) with the likelihood of each confusion dependingon the writer’s native language (a channel model). The channel model could be trainedon text written by second-language learners who speak, as a first language, the particularlanguage of interest.In the absence of such data, we only allow our system to make corrections in Englishif the proposed replacement shares a foreign-language translation in a particular Freelangonline bilingual dictionary (www.freelang.net/dictionary/). Put another way,we reduce the size of the preposition confusion set dynamically depending on the prepositionthat was used and the native language of the speaker. A particular preposition is onlysuggested as a correction if both the correction and the original preposition could have beenconfused by a foreign-language speaker translating a particular foreign-language prepositionwithout regard to context.To simulate the use of this module, we randomly flip 20% of our test-set prepositions toconfusable ones, and then apply our classifier with the aforementioned confusability (andconfidence) constraints. We experimented with French and German lexicons (Figure 3.2).These constraints strongly benefit both SUPERLM and TRIGRAM, with French constraints(−FR) helping slightly more than German (−DE) for higher coverage levels. There arefewer confusable prepositions in the French lexicon compared to German. As a baseline,if we assign our labels random scores, adding the French and German constraints resultsin 20% and 14% accuracy, respectively (compared to 134= 2.9% unconstrained). At 50%coverage, both constrained SUPERLM systems achieve close to 98% accuracy, a level thatcould provide very reliable feedback in second-language learning software.3.6 Context-Sensitive Spelling Correction3.6.1 The Task of Context-Sensitive Spelling CorrectionContext-sensitive spelling correction, or real-word error/malapropism detection [Goldingand Roth, 1999; Hirst and Budanitsky, 2005], is the task of identifying errors when a misspellingresults in a real word in the lexicon, e.g., using site when sight or cite was intended.Contextual spell checkers are among the most widely-used NLP technology, as they are includedin commercial word processing software [Church et al., 2007].For every occurrence of a word in a pre-defined confusion set (like{among, between}),46