Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub
14 # If there is a configuration file, builds a dictionary with15 # the corresponding start and end lines of each text file16 config_file = os.path.join(source, 'lines.cfg')17 config = {}18 if os.path.exists(config_file):19 with open(config_file, 'r') as f:20 rows = f.readlines()2122 for r in rows[1:]:23 fname, start, end = r.strip().split(',')24 config.update({fname: (int(start), int(end))})2526 new_fnames = []27 # For each file of text28 for fname in filenames:29 # If there's a start and end line for that file, use it30 try:31 start, end = config[fname]32 except KeyError:33 start = None34 end = None3536 # Opens the file, slices the configures lines (if any)37 # cleans line breaks and uses the sentence tokenizer38 with open(os.path.join(source, fname), 'r') as f:39 contents = (40 ''.join(f.readlines()[slice(start, end, None)])41 .replace('\n', ' ').replace('\r', '')42 )43 corpus = sent_tokenize(contents, **kwargs)4445 # Builds a CSV file containing tokenized sentences46 base = os.path.splitext(fname)[0]47 new_fname = f'{base}.sent.csv'48 new_fname = os.path.join(source, new_fname)49 with open(new_fname, 'w') as f:50 # Header of the file51 if include_header:52 if include_source:53 f.write('sentence,source\n')54 else:55 f.write('sentence\n')Building a Dataset | 889
56 # Writes one line for each sentence57 for sentence in corpus:58 if include_source:59 f.write(f'{quote_char}{sentence}{quote_char}\60 {sep_char}{fname}\n')61 else:62 f.write(f'{quote_char}{sentence}\63 {quote_char}\n')64 new_fnames.append(new_fname)6566 # Returns list of the newly generated CSV files67 return sorted(new_fnames)It takes a source folder (or a single file) and goes through the files with the rightextensions (only .txt by default), removing lines based on the lines.cfg file (ifany), applying the sentence tokenizer to each file, and generating thecorresponding CSV files of sentences using the configured quote_char andsep_char. It may also use include_header and include_source in the CSV file.The CSV files are named after the corresponding text files by dropping the originalextension and appending .sent.csv to it. Let’s see it in action:Generating Dataset of Sentences1 new_fnames = sentence_tokenize(localfolder)2 new_fnamesOutput['texts/alice28-1476.sent.csv', 'texts/wizoz10-1740.sent.csv']Each CSV file contains the sentences of a book, and we’ll use both of them to buildour own dataset.890 | Chapter 11: Down the Yellow Brick Rabbit Hole
- Page 864 and 865: • First, and most important, PyTo
- Page 866 and 867: decode(), with a single one, encode
- Page 868 and 869: 46 for i in range(self.target_len):
- Page 870 and 871: Figure 10.18 - Losses - PyTorch’s
- Page 872 and 873: Figure 10.20 - Sample image—label
- Page 874 and 875: 4041 # Builds a weighted random sam
- Page 876 and 877: Figure 10.23 - Sample image—split
- Page 878 and 879: Einops"There is more than one way t
- Page 880 and 881: Figure 10.26 - Two patch embeddings
- Page 882 and 883: Now each sequence has ten elements,
- Page 884 and 885: It takes an instance of a Transform
- Page 886 and 887: Putting It All TogetherIn this chap
- Page 888 and 889: 1. Encoder-DecoderThe encoder-decod
- Page 890 and 891: This is the actual encoder-decoder
- Page 892 and 893: 3. DecoderThe Transformer decoder h
- Page 894 and 895: 5. Encoder "Layer"The encoder "laye
- Page 896 and 897: 7. "Sub-Layer" WrapperThe "sub-laye
- Page 898 and 899: 8. Multi-Headed AttentionThe multi-
- Page 900 and 901: Model Configuration & TrainingModel
- Page 902 and 903: • training the Transformer to tac
- Page 904 and 905: Part IVNatural Language Processing|
- Page 906 and 907: Additional SetupThis is a special c
- Page 908 and 909: "Down the Yellow Brick Rabbit Hole"
- Page 910 and 911: The actual texts of the books are c
- Page 912 and 913: "What is this punkt?"That’s the P
- Page 916 and 917: Sentence Tokenization in spaCyBy th
- Page 918 and 919: AttributesThe Dataset has many attr
- Page 920 and 921: Output{'labels': 1,'sentence': 'The
- Page 922 and 923: elements from the text. But preproc
- Page 924 and 925: Data AugmentationLet’s briefly ad
- Page 926 and 927: The corpora’s dictionary is not a
- Page 928 and 929: Finally, if we want to convert a li
- Page 930 and 931: Once we’re happy with the size an
- Page 932 and 933: from transformers import BertTokeni
- Page 934 and 935: "What about the separation token?"T
- Page 936 and 937: The last output, attention_mask, wo
- Page 938 and 939: Outputtensor([[ 3, 27, 1, ..., 0, 0
- Page 940 and 941: vector, right? And our vocabulary i
- Page 942 and 943: Maybe you filled this blank in with
- Page 944 and 945: Continuous Bag-of-Words (CBoW)In th
- Page 946 and 947: That’s a fairly simple model, rig
- Page 948 and 949: Figure 11.13 - Continuous bag-of-wo
- Page 950 and 951: Figure 11.15 - Reviewing restaurant
- Page 952 and 953: You got that right—arithmetic—r
- Page 954 and 955: There we go, 50 dimensions! It’s
- Page 956 and 957: Equation 11.1 - Embedding arithmeti
- Page 958 and 959: Only 82 out of 50,802 words in the
- Page 960 and 961: Now we can use its encode() method
- Page 962 and 963: Model I — GloVE + ClassifierData
56 # Writes one line for each sentence
57 for sentence in corpus:
58 if include_source:
59 f.write(f'{quote_char}{sentence}{quote_char}\
60 {sep_char}{fname}\n')
61 else:
62 f.write(f'{quote_char}{sentence}\
63 {quote_char}\n')
64 new_fnames.append(new_fname)
65
66 # Returns list of the newly generated CSV files
67 return sorted(new_fnames)
It takes a source folder (or a single file) and goes through the files with the right
extensions (only .txt by default), removing lines based on the lines.cfg file (if
any), applying the sentence tokenizer to each file, and generating the
corresponding CSV files of sentences using the configured quote_char and
sep_char. It may also use include_header and include_source in the CSV file.
The CSV files are named after the corresponding text files by dropping the original
extension and appending .sent.csv to it. Let’s see it in action:
Generating Dataset of Sentences
1 new_fnames = sentence_tokenize(localfolder)
2 new_fnames
Output
['texts/alice28-1476.sent.csv', 'texts/wizoz10-1740.sent.csv']
Each CSV file contains the sentences of a book, and we’ll use both of them to build
our own dataset.
890 | Chapter 11: Down the Yellow Brick Rabbit Hole
Change language