www.allitebooks.com

Chapter 9
Character n-grams are found in text documents by representing the document as a
sequence of characters. These n-grams are then extracted from this sequence and a
model is trained. There are a number of different models for this, but a standard one
is very similar to the bag-of-words model we have used earlier.
For each distinct n-gram in the training corpus, we create a feature for it. An example
of an n-gram is , which is the letter e, a space, and then the letter t (the angle
brackets are used to denote the start and end of the n-gram and aren't part of it). We
then train our model using the frequency of each n-gram in the training documents
and train the classifier using the created feature matrix.
Character n-grams are defined in many ways. For instance,
some applications only choose within-word characters, ignoring
whitespace and punctuation. Some use this information (like our
implementation in this chapter).
A common theory for why character n-grams work is that people more typically
write words they can easily say and character n-grams (at least when n is between 2
and 6) are a good approximation for phonemes—the sounds we make when saying
words. In this sense, using character n-grams approximates the sounds of words,
which approximates your writing style. This is a common pattern when creating
new features. First we have a theory on what concepts will impact the end result
(authorship style) and then create features to approximate or measure those concepts.
A main feature of a character n-gram matrix is that it is sparse and increases
in sparsity with higher n-values quite quickly. For an n-value of 2, approximately 75
percent of our feature matrix is zeros. For an n-value of 5, over 93 percent is
zeros. This is typically less sparse than a word n-gram matrix of the same type
though and shouldn't cause many issues using a classifier that is used for
word-based classifications.
Extracting character n-grams
We are going to use our CountVectorizer class to extract character n-grams.
To do that, we set the analyzer parameter and specify a value for n to extract
n-grams with.
The implementation in scikit-learn uses an n-gram range, allowing you to extract
n-grams of multiple sizes at the same time. We won't delve into different n-values
in this experiment, so we just set the values the same. To extract n-grams of size 3,
you need to specify (3, 3) as the value for the n-gram range.
[ 199 ]