Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

The actual texts of the books are contained between lines 105 and 3703
(remember Python’s zero-based indexing) and 309 and 5099, respectively.
Moreover, we’re joining all the lines together into a single large string of text for
each book because we’re going to organize the resulting texts into sentences, and
in a regular book there are line breaks mid-sentence all over.
We definitely do not want to do that manually every time, right? Although it would
be more difficult to automatically remove any additions to the original text, we can
partially automate the removal of the extra lines by setting the real start and end
lines of each text in a configuration file (lines.cfg):
Configuration File
1 text_cfg = """fname,start,end
2 alice28-1476.txt,104,3704
3 wizoz10-1740.txt,310,5100"""
4 bytes_written = open(
5 os.path.join(localfolder, 'lines.cfg'), 'w'
6 ).write(text_cfg)
Your local folder (texts) should have three files now: alice28-1476.txt, lines.cfg,
and wizoz10-1740.txt. Now, it is time to perform…
Sentence Tokenization
A token is a piece of a text, and to tokenize a text means to split
it into pieces; that is, into a list of tokens.
"What kind of pieces are we talking about here?"
The most common kind of piece is a word. So, tokenizing a text usually means to
split it into words using the white space as a separator:
sentence = "I'm following the white rabbit"
tokens = sentence.split(' ')
tokens
Building a Dataset | 885