Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Then, let’s create an instance of the data collator and apply it to our mini-batch of
one:
from transformers import DataCollatorForLanguageModeling
torch.manual_seed(41)
data_collator = DataCollatorForLanguageModeling(
tokenizer=bert_tokenizer, mlm_probability=0.15
)
mlm_tokens = data_collator([tokens])
mlm_tokens
Output
{'input_ids': tensor([[ 101, 5650, 2003, 1999, 10288, 24759,
103, 6321, 2206, 1996, 2317, 10442, 102]]),
'labels': tensor([[-100, -100, -100, -100, -100, -100, 5555, -100,
-100, -100, -100, -100, -100]])}
If you look closely, you’ll see that the seventh token (5555 in the original input)
was replaced by some other token (103). Moreover, the labels contain the
replaced tokens in their original positions (and -100 everywhere else to indicate
these tokens are irrelevant for computing the loss). It’s actually easier to visualize
the difference if we convert the IDs back to tokens:
print(bert_tokenizer.convert_ids_to_tokens(tokens['input_ids']))
print(bert_tokenizer.convert_ids_to_tokens(
mlm_tokens['input_ids'][0]
))
Output
['[CLS]', 'alice', 'is', 'in', '##ex', '##pl', '##ica', '##bly',
'following', 'the', 'white', 'rabbit', '[SEP]']
['[CLS]', 'alice', 'is', 'in', '##ex', '##pl', '[MASK]', '##bly',
'following', 'the', 'white', 'rabbit', '[SEP]']
See? The seventh token (##ica) got masked!
BERT | 977