Large-Scale Semi-Supervised Learning for Natural Language ...

One way to find a better solution in this bumpy optimization space is to initialize orfix parameters of your model in ways that bias things toward what you know you want.For example, [Haghighi and Klein, 2010] fix a number of parameters in their entity-type /coreference model using prototypes of different classes. That is, they ensure, e.g., that Bushor Gore are in the PERSON class, as are the nominals president, official, etc., and thatthis class is referred to by the appropriate set of pronouns. They also set a number of otherparameters to “fixed heuristic values.” When the unsupervised learning kicks in, it initiallyhas less freedom to go off the rails, as the hand-tuning has started the model from a goodspot in the optimization space.One argument that is sometimes made against fully unsupervised approaches is that theset-up is a little unrealistic. You will likely want to evaluate your approach. To evaluateyour approach, you will need some labeled data. If you can produce labeled data for testing,you can produce some labeled data for training. It seems that semi-supervised learningis a more realistic situation: you have lots of unlabeled data, but you also have a few labeledexamples to help you configure your parameters. In our unsupervised pronoun resolutionwork [Cherry and Bergsma, 2005], we also used some labeled examples to re-weight the parameterslearned by EM (using the discriminative technique known as maximum entropy). 3Another interaction between unsupervised and supervised learning occurs when an unsupervisedmethod provides intermediate structural information for a supervised algorithm.For example, unsupervised algorithms often generate the unseen word-to-word alignmentsin statistical machine translation [Brown et al., 1993] and also the unseen character-tophonemealignments in grapheme-to-phoneme conversion [Jiampojamarn et al., 2007].This alignment information is then leveraged by subsequent supervised processing.2.5 Semi-Supervised LearningSemi-supervised learning is a huge and growing area of interest in many different researchcommunities. The name semi-supervised learning has come to essentially mean that apredictor is being created from information from both labeled and unlabeled examples.There are a variety of flavours of semi-supervised learning that are relevant to NLP andmerit discussion. A good recent survey of semi-supervised techniques in general is byZhu [2005].Semi-supervised learning was the “Special Topic of Interest” at the 2009 Conferenceon Natural Language Learning. The organizers, Suzanne Stevenson and Xavier Carreras,provided a thoughtful motivation for semi-supervised learning in the call for papers. 4“The field of natural language learning has made great strides over the last 15years, especially in the design and application of supervised and batch learningmethods. However, two challenges arise with this kind of approach. First, incore NLP tasks, supervised approaches require typically large amounts of manuallyannotated data, and experience has shown that results often depend on the3 The line between supervised and unsupervised learning can be a little blurry. We called our use of labeleddata and maximum entropy the “supervised extension” of our unsupervised system in our EM paper [Cherryand Bergsma, 2005]. A later unsupervised approach by Charniak and Elsner [2009], which also uses EMtraining for pronoun resolution, involved tuning essentially the same number of hyperparameters by hand (tooptimize performance on a development set) as the number of parameters we tuned with supervision. Is thisstill unsupervised learning?4 http://www.cnts.ua.ac.be/conll2009/cfp.html24