Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

"What about the separation token?"
This special token is used to, well, separate inputs into two distinct sentences. Yes,
it is possible to feed BERT with two sentences at once, and this kind of input is used
for the next sentence prediction task. We won’t be using that in our example, but
we’ll get back to it while discussing how BERT is trained.
We can actually get rid of the special tokens if we’re not using them:
tokenizer.encode(new_sentence, add_special_tokens=False)
Output
[1219, 5, 229, 200, 1]
"OK, but where is the promised additional information?"
That’s easy enough—we can simply call the tokenizer itself instead of a particular
method and it will produce an enriched output:
tokenizer(new_sentence,
add_special_tokens=False,
return_tensors='pt')
Output
{'input_ids': tensor([[1219, 5, 229, 200, 1]]),
'token_type_ids': tensor([[0, 0, 0, 0, 0]]),
'attention_mask': tensor([[1, 1, 1, 1, 1]])}
By default, the outputs are lists, but we used the return_tensors argument to get
PyTorch tensors instead (pt stands for PyTorch). There are three outputs in the
dictionary: input_ids, token_type_ids, and attention_mask.
The first one, input_ids, is the familiar list of token IDs. They are the most
fundamental input, and sometimes the only one, required by the model.
Word Tokenization | 909