Large-Scale Semi-Supervised Learning for Natural Language ...

Franz, 2006]. This corpus simply lists, for sequences of words from length two to lengthfive, how often the sequence occurs in their web corpus. The web corpus was generatedfrom approximately 1 trillion tokens of online text. In this data, tokens appearing less than200 times have been mapped to the 〈UNK〉 symbol. Also, only N-grams appearing morethan 40 times are included. A number of researchers have begun using this N-gram corpus,rather than search engines, to collect their web-scale statistics [Vadas and Curran, 2007a;Felice and Pulman, 2007; Yuret, 2007; Kummerfeld and Curran, 2008; Carlson et al., 2008;Bergsma et al., 2008b; Tratz and Hovy, 2010]. Although this N-gram data is much smallerthan the source text from which it was taken, it is still a very large resource, occupyingapproximately 24 GB compressed, and containing billions of N-grams in hundreds of files.Special strategies are needed to effectively query large numbers of counts. Some of thesestrategies include pre-sorting queries to reduce passes through the data, hashing [Hawkeret al., 2007], storing the data in a database [Carlson et al., 2008], and using a trie structure[Sekine, 2008]. Our work in this area led to our recent participation in the 2009 JohnsHopkins University, Center for Speech and Language Processing, Workshop on UnsupervisedAcquisition of Lexical Knowledge from N-Grams, led by Dekang Lin. 2 A number ofongoing projects using web-scale N-gram counts have arisen from this workshop, and wediscuss some of these in Chapter 5. Lin et al. [2010] provides an overview of our work atthe workshop, including the construction of a new web-scale N-gram corpus.In this chapter, all N-gram counts are taken from the standard Google N-gram data.One thing that N-gram data does not provide is the document co-occurrence counts thathave proven useful in some applications discussed above. It could therefore be beneficialto the community to have a resource along the lines of the Google N-gram corpus, butwhere the corpus simply states how often pairs of words (or phrases) co-occur within afixed window on the web. I am putting this on my to-do list.3.3 Disambiguation with N-gram CountsSection 3.2.1 described how lexical disambiguation, for both generation and analysis tasks,can be performed by scoring various context sequences using a statistical model. We formalizethe context used by web-scale systems and then discuss various statistical modelsthat use this information.For a word in text, v 0 , we wish to assign an output, c i , from a fixed set of candidates,C = {c 1 ,c 2 ...,c K }. Assume that our target word v 0 occurs in a sequence of context tokens:V={v −4 ,v −3 ,v −2 ,v −1 ,v 0 ,v 1 ,v 2 ,v 3 ,v 4 }. The key to improved web-scale modelsis that they make use of a variety of context segments, of different sizes and positions,that span the target word v 0 . We call these segments context patterns. The words that replacethe target word are called pattern fillers. Let the set of pattern fillers be denoted byF = {f 1 ,f 2 ,...,f |F| }. Recall that for generation tasks, the filler set will usually be identicalto the set of output candidates (e.g., for word selection tasks,F =C={among,between}). Foranalysis tasks, we must use other fillers, chosen as surrogates for one of the semantic labels(e.g. for WSD of bass,C={Sense1,Sense2},F={tenor,alto,pitch,snapper,mackerel,tuna}).Each length-N context pattern, with a filler in place of v 0 , is an N-gram, for whichwe can retrieve a count from an auxiliary corpus. We retrieve counts from the web-scaleGoogle Web 5-gram Corpus, which includes N-grams of length one to five (Section 3.2.2).2 http://www.clsp.jhu.edu/workshops/ws09/groups/ualkn/39