www.allitebooks.com

Working with Big Data
Getting the data
The data we are going to use is a set of blog posts that are labeled for age, gender,
industry (that is, work) and, funnily enough, star sign. This data was collected from
http://blogger.com in August 2004 and has over 140 million words in more than
600,000 posts. Each blog is probably written by just one person, with some work put
into verifying this (although, we can never be really sure). Posts are also matched
with the date of posting, making this a very rich dataset.
To get the data, go to http://u.cs.biu.ac.il/~koppel/BlogCorpus.htm and click
on Download Corpus. From there, unzip the file to a directory on your computer.
The dataset is organized with a single blog to a file, with the filename giving the
classes. For instance, one of the filenames is as follows:
1005545.male.25.Engineering.Sagittarius.xml
The filename is separated by periods, and the fields are as follows:
• Blogger ID: This a simple ID value to organize the identities.
• Gender: This is either male or female, and all the blogs are identified as one
of these two options (no other options are included in this dataset).
• Age: The exact ages are given, but some gaps are deliberately present. Ages
present are in the (inclusive) ranges of 13-17, 23-27, and 33-48. The reason
for the gaps is to allow for splitting the blogs into age ranges with gaps, as
it would be quite difficult to separate an 18 year old's writing from a 19 year
old, and it is possible that the age itself is a little outdated.
• Industry: In one of 40 different industries including science, engineering,
arts, and real estate. Also, included is indUnk, for unknown industry.
• Star Sign: This is one of the 12 astrological star signs.
All values are self-reported, meaning there may be errors or inconsistencies with
labeling, but are assumed to be mostly reliable—people had the option of not
setting values if they wanted to preserve their privacy in those ways.
A single file is in a psuedo-XML format, containing a tag and then a sequence
of tags. Each of the tag is proceeded by a tag as well. While
we can parse this as XML, it is much simpler to parse it on a line-by-line basis as the
files are not exactly well-formed XML, with some errors (mostly encoding problems).
To read the posts in the file, we can use a loop to iterate over the lines.
[ 280 ]