www.allitebooks.com

Chapter 12
python extract_posts.py -r emr s3://ch12gender/blogs_train_large/
--output-dir=s3://ch12/blogposts_train/ --no-output
python nb_train.py -r emr s3://ch12/blogposts_train/ --output-dir=s3://
ch12/model/ --o-output
You will also be charged for the usage. This will only be a few dollars,
but keep this in mind if you are going to keep running the jobs or doing
other jobs on bigger datasets. I ran a very large number of jobs and was
charged about $20 all up. Running just these few should be less than
$4. However, you can check your balance and set up pricing alerts, by
going to https://console.aws.amazon.com/billing/home.
It isn't necessary for the blogposts_train and model folders to exist—they will be
created by EMR. In fact, if they exist, you will get an error. If you are rerunning this,
just change the names of these folders to something new, but remember to change
both commands to the same names (that is, the output directory of the first command
is the input directory of the second command).
If you are getting impatient, you can always stop the first job after
a while and just use the training data gathered so far. I recommend
leaving the job for an absolute minimum of 15 minutes and probably
at least an hour. You can't stop the second job and get good results
though; the second job will probably take about two to three times as
long as the first job did.
You can now go back to the s3 console and download the output model from your
bucket. Saving it locally, we can go back to our IPython Notebook and use the new
model. We reenter the code here—only the differences are highlighted, just to update
to our new model:
aws_model_filename = os.path.join(os.path.expanduser("~"), "models",
"aws_model")
aws_model = load_model(aws_model_filename)
y_true = []
y_pred = []
for actual_gender, predicted_gender in nb_predict_many(aws_model,
testing_filenames[0]):
y_true.append(actual_gender == "female")
y_pred.append(predicted_gender == "female")
y_true = np.array(y_true, dtype='int')
y_pred = np.array(y_pred, dtype='int')
print("f1={:.4f}".format(f1_score(y_true, y_pred, pos_label=None)))
[ 295 ]