Large-Scale Semi-Supervised Learning for Natural Language ...

problems. However, they do not work well in all situations. Suppose we wanted a binaryclassifier to tell us whether a given Canadian city is either in Manitoba or not in Manitoba.Suppose we had only one feature: distance from the Pacific ocean. It would be difficult tochoose a weight and a threshold for a linear classifier such that we could separate Manitobancities from other Canadian cities with only this one feature. If we took cities below athreshold, we could get all cities west of Ontario. If we took those above, we could getall cities east of Saskatchewan. We would say that the positive and negative examples arenot separable using this feature representation; the positives and negatives can’t be placednicely onto either side of a hyperplane. There are lots of non-linear classifiers to choosefrom that might do better.On the other hand, we’re always free to choose whatever feature function we like; formost problems, we can just choose a feature space that does work well with linear classifiers(i.e., a feature space that perhaps does make the training data separable). We could dividedistance-from-Pacific-ocean into multiple features: say, a binary feature if the distance isbetween 0 and 100 km, another if it’s between 100 and 200, etc. Also, many learningalgorithms permit us to use the kernel trick, which maps the feature vectors into an implicithigher-dimensional space where a linear hyperplane can better divide the classes. We returnto this point briefly in the following section. For many natural language problems, we havethousands of relevant features, and good classification is possible with linear classifiers.Generally, the more features, the more separable the examples.2.3 Supervised LearningIn this section, we provide a very practical discussion of how the parameters of the linearclassifier are chosen. This is the NLP view of machine learning: what you need to know touse it as a tool.2.3.1 Experimental Set-upThe proper set-up is to have at least three sets of labeled data when designing a supervisedmachine learning system. First, you have a training set, which you use to learn your model(yet another word that means the same thing as the weights or parameters: the model is theset of weights). Secondly, you have a development set, which serves two roles: a) you canset any of your algorithm’s hyperparameters on this set (hyperparameters are discussed below),and b) you can test your system on this set as you are developing. Rather than havinga single development set, you could optimize your parameters by ten-fold cross validationon the training set, essentially re-using the training data to set development parameters.Finally, you have a hold-out set or test set of unseen data which you use for your finalevaluation. You only evaluate on the test set once, to generate the final results of your experimentsfor your paper. This simulates how your algorithm would actually be used inpractice: classifying data it has not seen before.To run machine learning in this framework, we typically begin by converting the threesets into feature vectors and labels. We then supply the training set, in labeled featurevector format, to a standard software package, and this package returns the weights. Thepackage can also be used to multiply the feature vectors by the weights, and return theclassification decisions for new examples. It thus can and often does calculate performanceon the development or test sets for you.17