Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Every word piece is prefixed with ## to indicate that it does not
stand on its own as a word.
Given enough word pieces in a vocabulary, it will be able to represent every
unknown word using a concatenation of word pieces. Problem solved! That’s what
BERT’s pre-trained tokenizer does.
For more details on the WordPiece tokenizer, as well as other
sub-word tokenizers like Byte-Pair Encoding (BPE) and
SentencePiece, please check HuggingFace’s "Summary of the
Tokenizers" [207] and Cathal Horan’s great post "Tokenizers: How
machines read" [208] on FloydHub.
Let’s tokenize a pair of sentences using BERT’s WordPiece tokenizer:
sentence1 = 'Alice is inexplicably following the white rabbit'
sentence2 = 'Follow the white rabbit, Neo'
tokens = bert_tokenizer(sentence1, sentence2, return_tensors='pt')
tokens
Output
{'input_ids': tensor([[ 101, 5650, 2003, 1999, 10288, 24759,
5555, 6321, 2206, 1996, 2317, 10442, 102, 3582, 1996, 2317,
10442, 1010, 9253, 102]]),
'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
1, 1, 1, 1, 1, 1, 1]]),
'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1]])}
Notice that, since there are two sentences, the token_type_ids have two distinct
values (zero and one) that work as the sentence index corresponding to the
sentence each token belongs to. Hold this thought, because we’re using this
information in the next section.
BERT | 969