22.02.2024
•
Views
model anyway. We’ll use it to compute statistics only. By the way, we needstatistics for each channel, as required by the Normalize() transform.So, let’s build a function that takes a mini-batch (images and labels) and computesthe mean pixel value and standard deviation per channel of each image, adding upthe results for all images. Better yet, let’s make it a method of our StepByStep classtoo.StepByStep Method@staticmethoddef statistics_per_channel(images, labels):# NCHWn_samples, n_channels, n_height, n_weight = images.size()# Flatten HW into a single dimensionflatten_per_channel = images.reshape(n_samples, n_channels, -1)# Computes statistics of each image per channel# Average pixel value per channel# (n_samples, n_channels)means = flatten_per_channel.mean(axis=2)# Standard deviation of pixel values per channel# (n_samples, n_channels)stds = flatten_per_channel.std(axis=2)# Adds up statistics of all images in a mini-batch# (1, n_channels)sum_means = means.sum(axis=0)sum_stds = stds.sum(axis=0)# Makes a tensor of shape (1, n_channels)# with the number of samples in the mini-batchn_samples = torch.tensor([n_samples]*n_channels).float()# Stack the three tensors on top of one another# (3, n_channels)return torch.stack([n_samples, sum_means, sum_stds], axis=0)setattr(StepByStep, 'statistics_per_channel',statistics_per_channel)Data Preparation | 421
first_images, first_labels = next(iter(temp_loader))StepByStep.statistics_per_channel(first_images, first_labels)Outputtensor([[16.0000, 16.0000, 16.0000],[13.8748, 13.3048, 13.1962],[ 3.0507, 3.8268, 3.9754]])Applying it to the first mini-batch of images, we get the results above: Each columnrepresents a channel, and the rows are the number of data points, the sum of meanvalues, and the sum of standard deviations, respectively.We can leverage the loader_apply() method we created in the last chapter to getthe sums for the whole dataset:results = StepByStep.loader_apply(temp_loader,StepByStep.statistics_per_channel)resultsOutputtensor([[2520.0000, 2520.0000, 2520.0000],[2142.5359, 2070.0811, 2045.1442],[ 526.3024, 633.0677, 669.9554]])So, we can compute the average mean value (that sounds weird, I know) and theaverage standard deviation, per channel. Better yet, let’s make it a method thattakes a data loader and returns an instance of the Normalize() transform,statistics and all:422 | Chapter 6: Rock, Paper, Scissors
-
Page 2 and 3:
Deep Learning with PyTorchStep-by-S
-
Page 4 and 5:
"What I cannot create, I do not und
-
Page 6 and 7:
Train-Validation-Test Split. . . .
-
Page 8 and 9:
Random Split. . . . . . . . . . . .
-
Page 10 and 11:
The Precision Quirk . . . . . . . .
-
Page 12 and 13:
A REAL Filter . . . . . . . . . . .
-
Page 14 and 15:
Jupyter Notebook . . . . . . . . .
-
Page 16 and 17:
Stacked RNN . . . . . . . . . . . .
-
Page 18 and 19:
Attention Mechanism . . . . . . . .
-
Page 20 and 21:
4. Positional Encoding. . . . . . .
-
Page 22 and 23:
Model Configuration & Training . .
-
Page 24 and 25:
AcknowledgementsFirst and foremost,
-
Page 26 and 27:
Frequently Asked Questions (FAQ)Why
-
Page 28 and 29:
There is yet another advantage of f
-
Page 30 and 31:
• Classes and methods are written
-
Page 32 and 33:
What’s Next?It’s time to set up
-
Page 34 and 35:
After choosing a repository, it wil
-
Page 36 and 37:
1. AnacondaIf you don’t have Anac
-
Page 38 and 39:
3. PyTorchPyTorch is the coolest de
-
Page 40 and 41:
(pytorchbook) C:\> conda install py
-
Page 42 and 43:
(pytorchbook)$ pip install torchviz
-
Page 44 and 45:
7. JupyterAfter cloning the reposit
-
Page 46 and 47:
Part IFundamentals| 21
-
Page 48 and 49:
notebook. If not, just click on Cha
-
Page 50 and 51:
Also, let’s say that, on average,
-
Page 52 and 53:
There is one exception to the "alwa
-
Page 54 and 55:
Random Initialization1 # Step 0 - I
-
Page 56 and 57:
Batch, Mini-batch, and Stochastic G
-
Page 58 and 59:
Outputarray([[-2. , -1.94, -1.88, .
-
Page 60 and 61:
one matrix for each data point, eac
-
Page 62 and 63:
Sure, different values of b produce
-
Page 64 and 65:
Output-3.044811379650508 -1.8337537
-
Page 66 and 67:
each parameter using the chain rule
-
Page 68 and 69:
What’s the impact of one update o
-
Page 70 and 71:
gradients, we know we need to take
-
Page 72 and 73:
Very High Learning RateWait, it may
-
Page 74 and 75:
true_b = 1true_w = 2N = 100# Data G
-
Page 76 and 77:
Let’s look at the cross-sections
-
Page 78 and 79:
Zero Mean and Unit Standard Deviati
-
Page 80 and 81:
Sure, in the real world, you’ll n
-
Page 82 and 83:
computing the loss, as shown in the
-
Page 84 and 85:
• visualizing the effects of usin
-
Page 86 and 87:
If you’re using Jupyter’s defau
-
Page 88 and 89:
Notebook Cell 1.1 - Splitting synth
-
Page 90 and 91:
Step 2# Step 2 - Computing the loss
-
Page 92 and 93:
Output[0.49671415] [-0.1382643][0.8
-
Page 94 and 95:
Notebook Cell 1.2 - Implementing gr
-
Page 96 and 97:
# Sanity Check: do we get the same
-
Page 98 and 99:
Outputtensor(3.1416)tensor([1, 2, 3
-
Page 100 and 101:
Outputtensor([[1., 2., 1.],[1., 1.,
-
Page 102 and 103:
dummy_array = np.array([1, 2, 3])du
-
Page 104 and 105:
n_cudas = torch.cuda.device_count()
-
Page 106 and 107:
back_to_numpy = x_train_tensor.nump
-
Page 108 and 109:
I am assuming you’d like to use y
-
Page 110 and 111:
Outputtensor([0.1940], device='cuda
-
Page 112 and 113:
print(error.requires_grad, yhat.req
-
Page 114 and 115:
Output(tensor([0.], device='cuda:0'
-
Page 116 and 117:
56 # need to tell it to let it go..
-
Page 118 and 119:
computation.If you chose "Local Ins
-
Page 120 and 121:
Figure 1.6 - Now parameter "b" does
-
Page 122 and 123:
There are many optimizers: SGD is t
-
Page 124 and 125:
41 optimizer.zero_grad() 34243 prin
-
Page 126 and 127:
Notebook Cell 1.8 - PyTorch’s los
-
Page 128 and 129:
Outputarray(0.00804466, dtype=float
-
Page 130 and 131:
Let’s build a proper (yet simple)
-
Page 132 and 133:
"What do we need this for?"It turns
-
Page 134 and 135:
1 Instantiating a model2 What IS th
-
Page 136 and 137:
In the __init__() method, we create
-
Page 138 and 139:
LayersA Linear model can be seen as
-
Page 140 and 141:
There are MANY different layers tha
-
Page 142 and 143:
We use magic, just like that:%run -
-
Page 144 and 145:
• Step 1: compute model’s predi
-
Page 146 and 147:
RecapFirst of all, congratulations
-
Page 148 and 149:
Chapter 2Rethinking the Training Lo
-
Page 150 and 151:
Let’s take a look at the code onc
-
Page 152 and 153:
Higher-Order FunctionsAlthough this
-
Page 154 and 155:
def exponentiation_builder(exponent
-
Page 156 and 157:
Apart from returning the loss value
-
Page 158 and 159:
Our code should look like this; see
-
Page 160 and 161:
There is no need to load the whole
-
Page 162 and 163:
but if we want to get serious about
-
Page 164 and 165:
How does this change our code so fa
-
Page 166 and 167:
Run - Model Training V2%run -i mode
-
Page 168 and 169:
piece of code that’s going to be
-
Page 170 and 171:
for it. We could do the same for th
-
Page 172 and 173:
EvaluationHow can we evaluate the m
-
Page 174 and 175:
And then, we update our model confi
-
Page 176 and 177:
Run - Model Training V4%run -i mode
-
Page 178 and 179:
Loading Extension# Load the TensorB
-
Page 180 and 181:
browser, you’ll likely see someth
-
Page 182 and 183:
model’s graph (not quite the same
-
Page 184 and 185:
Figure 2.5 - Scalars on TensorBoard
-
Page 186 and 187:
Define - Model Training V51 %%write
-
Page 188 and 189:
If, by any chance, you ended up wit
-
Page 190 and 191:
The procedure is exactly the same,
-
Page 192 and 193:
soon, so please bear with me for no
-
Page 194 and 195:
After recovering our model’s stat
-
Page 196 and 197:
Run - Model Configuration V31 # %lo
-
Page 198 and 199:
This is the general structure you
-
Page 200 and 201:
Chapter 2.1Going ClassySpoilersIn t
-
Page 202 and 203:
# A completely empty (and useless)
-
Page 204 and 205:
# These attributes are defined here
-
Page 206 and 207:
# Creates the train_step function f
-
Page 208 and 209:
# Builds function that performs a s
-
Page 210 and 211:
setattrThe setattr function sets th
-
Page 212 and 213:
See? We effectively modified the un
-
Page 214 and 215:
the random seed as arguments.This s
-
Page 216 and 217:
The current state of development of
-
Page 218 and 219:
Lossesdef plot_losses(self):fig = p
-
Page 220 and 221:
Run - Data Preparation V21 # %load
-
Page 222 and 223:
Model TrainingWe start by instantia
-
Page 224 and 225:
Making PredictionsLet’s make up s
-
Page 226 and 227:
OutputOrderedDict([('0.weight', ten
-
Page 228 and 229:
Run - Data Preparation V21 # %load
-
Page 230 and 231:
• defining our StepByStep class
-
Page 232 and 233:
import numpy as npimport torchimpor
-
Page 234 and 235:
Next, we’ll standardize the featu
-
Page 236 and 237:
Equation 3.1 - A linear regression
-
Page 238 and 239:
The odds ratio is given by the rati
-
Page 240 and 241:
As expected, probabilities that add
-
Page 242 and 243:
Sigmoid Functiondef sigmoid(z):retu
-
Page 244 and 245:
A picture is worth a thousand words
-
Page 246 and 247:
OutputOrderedDict([('linear.weight'
-
Page 248 and 249:
The first summation adds up the err
-
Page 250 and 251:
IMPORTANT: Make sure to pass the pr
-
Page 252 and 253:
To make it clear: In this chapter,
-
Page 254 and 255:
argument of nn.BCEWithLogitsLoss().
-
Page 256 and 257:
It is not that hard, to be honest.
-
Page 258 and 259:
Figure 3.6 - Training and validatio
-
Page 260 and 261:
Outputarray([[0.5504593 ],[0.949995
-
Page 262 and 263:
decision boundary.Look at the expre
-
Page 264 and 265:
Are my data points separable?That
-
Page 266 and 267:
model = nn.Sequential()model.add_mo
-
Page 268 and 269:
It looks like this:Figure 3.10 - Sp
-
Page 270 and 271:
True and False Positives and Negati
-
Page 272 and 273:
tpr_fpr(cm_thresh50)Output(0.909090
-
Page 274 and 275:
The trade-off between precision and
-
Page 276 and 277:
Figure 3.13 - Using a low threshold
-
Page 278 and 279:
Figure 3.16 - Trade-offs for two di
-
Page 280 and 281:
thresholds do not necessarily inclu
-
Page 282 and 283:
actual data, it is as bad as it can
-
Page 284 and 285:
If you want to learn more about bot
-
Page 286 and 287:
Model Training1 n_epochs = 10023 sb
-
Page 288 and 289:
step in your journey! What’s next
-
Page 290 and 291:
Chapter 4Classifying ImagesSpoilers
-
Page 292 and 293:
Data GenerationOur images are quite
-
Page 294 and 295:
Images and ChannelsIn case you’re
-
Page 296 and 297:
image_rgb = np.stack([image_r, imag
-
Page 298 and 299:
That’s fairly straightforward; we
-
Page 300 and 301:
• Transformations based on Tensor
-
Page 302 and 303:
position of an object in a picture
-
Page 304 and 305:
Outputtensor([[[0., 0., 0., 1., 0.]
-
Page 306 and 307:
Outputtensor([[[-1., -1., -1., 1.,
-
Page 308 and 309:
We can convert the former into the
-
Page 310 and 311:
composer = Compose([RandomHorizonta
-
Page 312 and 313:
Output<torch.utils.data.dataset.Sub
-
Page 314 and 315:
train_composer = Compose([RandomHor
-
Page 316 and 317:
The minority class should have the
-
Page 318 and 319:
train_loader = DataLoader(dataset=t
-
Page 320 and 321:
implemented in Chapter 2.1? Let’s
-
Page 322 and 323:
Let’s take one mini-batch of imag
-
Page 324 and 325:
What does our model look like? Visu
-
Page 326 and 327:
Model TrainingLet’s train our mod
-
Page 328 and 329:
preceding hidden layer to compute i
-
Page 330 and 331:
fig = sbs_nn.plot_losses()Figure 4.
-
Page 332 and 333:
Equation 4.2 - Equivalence of deep
-
Page 334 and 335:
w_nn_equiv = w_nn_output.mm(w_nn_hi
-
Page 336 and 337:
Weights as PixelsDuring data prepar
-
Page 338 and 339:
is only 0.25 (for z = 0) and that i
-
Page 340 and 341:
nn.Tanh()(dummy_z)Outputtensor([-0.
-
Page 342 and 343:
dummy_z = torch.tensor([-3., 0., 3.
-
Page 344 and 345:
As you can see, in PyTorch the coef
-
Page 346 and 347:
Figure 4.16 - Deep model (for real)
-
Page 348 and 349:
Figure 4.18 - Losses (before and af
-
Page 350 and 351:
Equation 4.3 - Activation functions
-
Page 352 and 353:
Helper Function #41 def index_split
-
Page 354 and 355:
Model Configuration1 # Sets learnin
-
Page 356 and 357:
Bonus ChapterFeature SpaceThis chap
-
Page 358 and 359:
Affine TransformationsAn affine tra
-
Page 360 and 361:
Figure B.3 - Annotated model diagra
-
Page 362 and 363:
Figure B.5 - In the beginning…But
-
Page 364 and 365:
OK, now we can clearly see a differ
-
Page 366 and 367:
In the model above, the sigmoid fun
-
Page 368 and 369:
the more dimensions, the more separ
-
Page 370 and 371:
import randomimport numpy as npfrom
-
Page 372 and 373:
identity = np.array([[[[0, 0, 0],[0
-
Page 374 and 375:
Figure 5.4 - Striding the image, on
-
Page 376 and 377:
Output-----------------------------
-
Page 378 and 379:
Outputtensor([[[[9., 5., 0., 7.],[0
-
Page 380 and 381:
OutputParameter containing:tensor([
-
Page 382 and 383:
Moreover, notice that if we were to
-
Page 384 and 385:
In code, as usual, PyTorch gives us
-
Page 386 and 387:
Outputtensor([[[[5., 5., 0., 8., 7.
-
Page 388 and 389:
edge = np.array([[[[0, 1, 0],[1, -4
-
Page 390 and 391:
A pooling kernel of two-by-two resu
-
Page 392 and 393:
Outputtensor([[22., 23., 11., 24.,
-
Page 394 and 395:
Figure 5.15 - LeNet-5 architectureS
-
Page 396 and 397:
• second block: produces 16-chann
-
Page 398 and 399:
Transformed Dataset1 class Transfor
-
Page 400 and 401:
LossNew problem, new loss. Since we
-
Page 402 and 403:
Outputtensor([4.0000, 1.0000, 0.500
-
Page 404 and 405:
The loss only considers the predict
-
Page 406 and 407:
Outputtensor([[-1.5229, -0.3146, -2
-
Page 408 and 409:
IMPORTANT: I can’t stress this en
-
Page 410 and 411:
figures at the beginning of this ch
-
Page 412 and 413:
The three units in the output layer
-
Page 414 and 415:
StepByStep Method@staticmethoddef _
-
Page 416 and 417:
The meow() method is totally indepe
-
Page 418 and 419:
StepByStep Methoddef visualize_filt
-
Page 420 and 421:
dummy_model = nn.Linear(1, 1)dummy_
-
Page 422 and 423:
dummy_listOutput[(Linear(in_feature
-
Page 424 and 425:
Output{Conv2d(1, 1, kernel_size=(3,
-
Page 426 and 427:
will be the externally defined vari
-
Page 428 and 429:
Removing Hookssbs_cnn1.remove_hooks
-
Page 430 and 431:
return figsetattr(StepByStep, 'visu
-
Page 432 and 433:
Figure 5.22 - Feature maps (classif
-
Page 434 and 435:
classification: The predicted class
-
Page 436 and 437:
convolutional layers to our model a
-
Page 438 and 439:
Capturing Outputsfeaturizer_layers
-
Page 440 and 441:
the filters learned by the model pr
-
Page 442 and 443:
given chapter are imported at its v
-
Page 444 and 445:
Data PreparationThe data preparatio
-
Page 448 and 449:
StepByStep Method@staticmethoddef m
-
Page 450 and 451:
"What’s wrong with the colors?"Th
-
Page 452 and 453:
three_channel_filter = np.array([[[
-
Page 454 and 455:
Fancier Model (Constructor)class CN
-
Page 456 and 457:
Fancier Model (Classifier)def class
-
Page 458 and 459:
torch.manual_seed(44)dropping_model
-
Page 460 and 461:
Outputtensor([0.1000, 0.2000, 0.300
-
Page 462 and 463:
Figure 6.8 - Output distribution fo
-
Page 464 and 465:
Adaptive moment estimation (Adam) u
-
Page 466 and 467:
torch.manual_seed(13)# Model Config
-
Page 468 and 469:
Outputtorch.Size([5, 3, 3, 3])Its s
-
Page 470 and 471:
Choosing a learning rate that works
-
Page 472 and 473:
Higher-Order Learning Rate Function
-
Page 474 and 475:
Perfect! Now let’s build the actu
-
Page 476 and 477:
ax.set_xlabel('Learning Rate')ax.se
-
Page 478 and 479:
LRFinderThe function we’ve implem
-
Page 480 and 481:
value in our moving average has an
-
Page 482 and 483:
Figure 6.15 - Distribution of weigh
-
Page 484 and 485:
In code, the implementation of the
-
Page 486 and 487:
As expected, the EWMA without corre
-
Page 488 and 489:
optimizer = optim.Adam(model.parame
-
Page 490 and 491:
IMPORTANT: The logging function mus
-
Page 492 and 493:
Output{'state': {140601337662512: {
-
Page 494 and 495:
different optimizer, set them to ca
-
Page 496 and 497:
• dampening: dampening factor for
-
Page 498 and 499:
Figure 6.20 - Paths taken by SGD (w
-
Page 500 and 501:
Equation 6.16 - Looking aheadOnce N
-
Page 502 and 503:
Figure 6.22 - Path taken by each SG
-
Page 504 and 505:
for epoch in range(4):# training lo
-
Page 506 and 507:
course) up to a given number of epo
-
Page 508 and 509:
Next, we create a protected method
-
Page 510 and 511:
Mini-Batch SchedulersThese schedule
-
Page 512 and 513:
Schedulers in StepByStep — Part I
-
Page 514 and 515:
Scheduler PathsBefore trying out a
-
Page 516 and 517:
After applying each scheduler to SG
-
Page 518 and 519:
Data Preparation1 # Loads temporary
-
Page 520 and 521:
Figure 6.31 - LossesEvaluationprint
-
Page 522 and 523:
[96] http://www.samkass.com/theorie
-
Page 524 and 525:
ImportsFor the sake of organization
-
Page 526 and 527:
ILSVRC-2012The 2012 edition [111] o
-
Page 528 and 529:
remained unchanged.ResNet (MSRA Tea
-
Page 530 and 531:
Transfer Learning in PracticeIn Cha
-
Page 532 and 533:
dropout. You’re already familiar
-
Page 534 and 535:
OutputDownloading: "https://downloa
-
Page 536 and 537:
Replacing the "Top" of the Model1 a
-
Page 538 and 539:
Model Size Classifier Layer(s) Repl
-
Page 540 and 541:
Model TrainingWe have everything se
-
Page 542 and 543:
"Removing" the Top Layer1 alex.clas
-
Page 544 and 545:
torch.save(train_preproc.tensors, '
-
Page 546 and 547:
Outputtensor([[109, 124],[124, 124]
-
Page 548 and 549:
Model Configuration1 optimizer_mode
-
Page 550 and 551:
Figure 7.4 - 1x1 convolutionThe inp
-
Page 552 and 553:
The weights used by PIL are 0.299 f
-
Page 554 and 555:
• reduce the number of output cha
-
Page 556 and 557:
The constructor method defines the
-
Page 558 and 559:
Does it sound familiar? That’s wh
-
Page 560 and 561:
and w to represent these parameters
-
Page 562 and 563:
A mini-batch of size 64 is small en
-
Page 564 and 565:
normed1 = batch_normalizer(batch1[0
-
Page 566 and 567:
OutputOrderedDict([('running_mean',
-
Page 568 and 569:
OutputOrderedDict([('running_mean',
-
Page 570 and 571:
batch_normalizer = nn.BatchNorm2d(n
-
Page 572 and 573:
torch.manual_seed(23)dummy_points =
-
Page 574 and 575:
np.concatenate([dummy_points[:5].nu
-
Page 576 and 577:
Another advantage of these shortcut
-
Page 578 and 579:
It should be pretty clear, except f
-
Page 580 and 581:
Data Preparation1 # ImageNet statis
-
Page 582 and 583:
Data Preparation — Preprocessing1
-
Page 584 and 585:
• freezing the layers of the mode
-
Page 586 and 587:
Extra ChapterVanishing and Explodin
-
Page 588 and 589:
discussing it, let me illustrate it
-
Page 590 and 591:
Model Configuration (2)1 loss_fn =
-
Page 592 and 593:
weights. If done properly, the init
-
Page 594 and 595:
just did), or, if you are training
-
Page 596 and 597:
Figure E.3 - The effect of batch no
-
Page 598 and 599:
Model Configuration1 torch.manual_s
-
Page 600 and 601:
torch.manual_seed(42)parm = nn.Para
-
Page 602 and 603:
(and only if) the norm exceeds the
-
Page 604 and 605:
if callable(self.clipping): 1self.c
-
Page 606 and 607:
Moreover, let’s use a ten times h
-
Page 608 and 609:
Clipping with HooksFirst, we reset
-
Page 610 and 611:
• visualizing the difference betw
-
Page 612 and 613:
Chapter 8SequencesSpoilersIn this c
-
Page 614 and 615:
Before shuffling, the pixels were o
-
Page 616 and 617:
And then let’s visualize the firs
-
Page 618 and 619:
sequence so far, and a data point f
-
Page 620 and 621:
Considering this, the not "unrolled
-
Page 622 and 623:
linear_input = nn.Linear(n_features
-
Page 624 and 625:
Outputtensor([[0.3924, 0.8146]], gr
-
Page 626 and 627:
Now we’re talking! The last hidde
-
Page 628 and 629:
Let’s take a look at the RNN’s
-
Page 630 and 631:
◦ The initial hidden state, which
-
Page 632 and 633:
batch_first argument to True so we
-
Page 634 and 635:
OutputOrderedDict([('weight_ih_l0',
-
Page 636 and 637:
out, hidden = rnn_stacked(x)out, hi
-
Page 638 and 639:
_l0_reverse).Once again, let’s cr
-
Page 640 and 641:
For bidirectional RNNs, the last el
-
Page 642 and 643:
Model Configuration1 class SquareMo
-
Page 644 and 645:
StepByStep.loader_apply(test_loader
-
Page 646 and 647:
Figure 8.14 - Final hidden states f
-
Page 648 and 649:
Figure 8.16 - Transforming the hidd
-
Page 650 and 651:
Since the RNN cell has both of them
-
Page 652 and 653:
Every gate worthy of its name will
-
Page 654 and 655:
• For r=0 and z=0, the cell becom
-
Page 656 and 657:
In code, we can use split() to get
-
Page 658 and 659:
Let’s pause for a moment here. Fi
-
Page 660 and 661:
Square Model II — The QuickeningT
-
Page 662 and 663:
Outputtensor([[53, 53],[75, 75]])Th
-
Page 664 and 665:
Figure 8.22 - Transforming the hidd
-
Page 666 and 667:
Equation 8.9 - LSTM—candidate hid
-
Page 668 and 669:
Now, let’s visualize the internal
-
Page 670 and 671:
OutputOrderedDict([('weight_ih', te
-
Page 672 and 673:
def forget_gate(h, x):thf = f_hidde
-
Page 674 and 675:
Outputtensor([[-5.4936e-02, -8.3816
-
Page 676 and 677:
1 First change: from RNN to LSTM2 S
-
Page 678 and 679:
Like the GRU, the LSTM presents fou
-
Page 680 and 681:
Output-----------------------------
-
Page 682 and 683:
Before moving on to packed sequence
-
Page 684 and 685:
column-wise fashion, from top to bo
-
Page 686 and 687:
does match the last output.• No,
-
Page 688 and 689:
So, to actually get the last output
-
Page 690 and 691:
Data Preparation1 class CustomDatas
-
Page 692 and 693:
OutputPackedSequence(data=tensor([[
-
Page 694 and 695:
Model Configuration & TrainingWe ca
-
Page 696 and 697:
size = 5weight = torch.ones(size) *
-
Page 698 and 699:
torch.manual_seed(17)conv_seq = nn.
-
Page 700 and 701:
Figure 8.32 - Applying dilated filt
-
Page 702 and 703:
Model Configuration1 torch.manual_s
-
Page 704 and 705:
We can actually find an expression
-
Page 706 and 707:
Data Preparation1 def pack_collate(
-
Page 708 and 709:
and variable-length sequences.Model
-
Page 710 and 711:
• generating variable-length sequ
-
Page 712 and 713:
import copyimport numpy as npimport
-
Page 714 and 715:
Figure 9.3 - Sequence datasetThe co
-
Page 716 and 717:
coordinates of a "perfect" square a
-
Page 718 and 719:
Let’s pretend for a moment that t
-
Page 720 and 721:
to initialize the hidden state and
-
Page 722 and 723:
predictions in previous steps have
-
Page 724 and 725:
the second set of predicted coordin
-
Page 726 and 727:
Let’s create an instance of the m
-
Page 728 and 729:
Model Configuration & TrainingThe m
-
Page 730 and 731:
Sure, we can!AttentionHere is a (no
-
Page 732 and 733:
based on "the" and "zone," I’ve j
-
Page 734 and 735:
Figure 9.12 - Matching a query to t
-
Page 736 and 737:
Outputtensor([[[ 0.0832, -0.0356],[
-
Page 738 and 739:
utmost importance for the correct i
-
Page 740 and 741:
Its formula is:Equation 9.3 - Cosin
-
Page 742 and 743:
second hidden state contributes to
-
Page 744 and 745:
Outputtensor([[[ 0.5475, 0.0875, -1
-
Page 746 and 747:
alphas = F.softmax(scaled_products,
-
Page 748 and 749:
Outputtensor([[[ 0.2138, -0.3175]]]
-
Page 750 and 751:
Attention Mechanism1 class Attentio
-
Page 752 and 753:
"Why would I want to force it to do
-
Page 754 and 755:
1 Sets attention module and adjusts
-
Page 756 and 757:
encdec = EncoderDecoder(encoder, de
-
Page 758 and 759:
fig = sbs_seq_attn.plot_losses()Fig
-
Page 760 and 761:
Figure 9.20 - Attention scoresSee?
-
Page 762 and 763:
Wide vs Narrow AttentionThis mechan
-
Page 764 and 765:
"What’s so special about it?"Even
-
Page 766 and 767:
Once again, the affine transformati
-
Page 768 and 769:
Next, we shift our focus to the sel
-
Page 770 and 771:
Encoder + Self-Attention1 class Enc
-
Page 772 and 773:
Figure 9.27 - Encoder with self- an
-
Page 774 and 775:
The figure below depicts the self-a
-
Page 776 and 777:
shifted_seq = torch.cat([source_seq
-
Page 778 and 779:
Equation 9.17 - Decoder’s (masked
-
Page 780 and 781:
At evaluation / prediction time we
-
Page 782 and 783:
Outputtensor([[[0.4132, 0.3728],[0.
-
Page 784 and 785:
Figure 9.33 - Encoder + decoder + a
-
Page 786 and 787:
64 return outputsThe encoder-decode
-
Page 788 and 789:
Figure 9.34 - Losses—encoder + de
-
Page 790 and 791:
curse. On the one hand, it makes co
-
Page 792 and 793:
"Are we done now? Is this good enou
-
Page 794 and 795:
Figure 9.46 - Consistent distancesA
-
Page 796 and 797:
Let’s recap what we’ve already
-
Page 798 and 799:
Let’s see it in code:max_len = 10
-
Page 800 and 801:
Let’s put it all together into a
-
Page 802 and 803:
Outputtensor([[[-1.0000, 0.0000],[-
-
Page 804 and 805:
Decoder with Positional Encoding1 c
-
Page 806 and 807:
Visualizing PredictionsLet’s plot
-
Page 808 and 809:
Next, we’re moving on to the thre
-
Page 810 and 811:
Data Generation & Preparation1 # Tr
-
Page 812 and 813:
59 self.trg_masks)60 else:61 # Deco
-
Page 814 and 815:
Model Configuration1 class EncoderS
-
Page 816 and 817:
1617 @property18 def alphas(self):1
-
Page 818 and 819:
Output(0.016193246061448008, 0.0341
-
Page 820 and 821:
sequential order of the data• fig
-
Page 822 and 823:
following imports:import copyimport
-
Page 824 and 825:
Figure 10.2 - Chunking: the wrong a
-
Page 826 and 827:
chunks to compute the other half of
-
Page 828 and 829:
67 # N, L, n_heads, d_k68 context =
-
Page 830 and 831:
dummy_points = torch.randn(16, 2, 4
-
Page 832 and 833:
Stacking Encoders and DecodersLet
-
Page 834 and 835:
"… with great depth comes great c
-
Page 836 and 837:
Transformer EncoderWe’ll be repre
-
Page 838 and 839:
Let’s see it in code, starting wi
-
Page 840 and 841:
Transformer Encoder1 class EncoderT
-
Page 842 and 843:
of the encoder-decoder (or Transfor
-
Page 844 and 845:
In PyTorch, the decoder "layer" is
-
Page 846 and 847:
In PyTorch, the decoder is implemen
-
Page 848 and 849:
Equation 10.7 - Data points' means
-
Page 850 and 851:
layer_norm = nn.LayerNorm(d_model)n
-
Page 852 and 853:
Figure 10.10 - Layer norm vs batch
-
Page 854 and 855:
Outputtensor([[[ 1.4636, 2.3663],[
-
Page 856 and 857:
The TransformerLet’s start with t
-
Page 858 and 859:
"values") in the decoder.• decode
-
Page 860 and 861:
Data Preparation1 # Generating trai
-
Page 862 and 863:
Figure 10.15 - Losses—Transformer
-
Page 864 and 865:
• First, and most important, PyTo
-
Page 866 and 867:
decode(), with a single one, encode
-
Page 868 and 869:
46 for i in range(self.target_len):
-
Page 870 and 871:
Figure 10.18 - Losses - PyTorch’s
-
Page 872 and 873:
Figure 10.20 - Sample image—label
-
Page 874 and 875:
4041 # Builds a weighted random sam
-
Page 876 and 877:
Figure 10.23 - Sample image—split
-
Page 878 and 879:
Einops"There is more than one way t
-
Page 880 and 881:
Figure 10.26 - Two patch embeddings
-
Page 882 and 883:
Now each sequence has ten elements,
-
Page 884 and 885:
It takes an instance of a Transform
-
Page 886 and 887:
Putting It All TogetherIn this chap
-
Page 888 and 889:
1. Encoder-DecoderThe encoder-decod
-
Page 890 and 891:
This is the actual encoder-decoder
-
Page 892 and 893:
3. DecoderThe Transformer decoder h
-
Page 894 and 895:
5. Encoder "Layer"The encoder "laye
-
Page 896 and 897:
7. "Sub-Layer" WrapperThe "sub-laye
-
Page 898 and 899:
8. Multi-Headed AttentionThe multi-
-
Page 900 and 901:
Model Configuration & TrainingModel
-
Page 902 and 903:
• training the Transformer to tac
-
Page 904 and 905:
Part IVNatural Language Processing|
-
Page 906 and 907:
Additional SetupThis is a special c
-
Page 908 and 909:
"Down the Yellow Brick Rabbit Hole"
-
Page 910 and 911:
The actual texts of the books are c
-
Page 912 and 913:
"What is this punkt?"That’s the P
-
Page 914 and 915:
14 # If there is a configuration fi
-
Page 916 and 917:
Sentence Tokenization in spaCyBy th
-
Page 918 and 919:
AttributesThe Dataset has many attr
-
Page 920 and 921:
Output{'labels': 1,'sentence': 'The
-
Page 922 and 923:
elements from the text. But preproc
-
Page 924 and 925:
Data AugmentationLet’s briefly ad
-
Page 926 and 927:
The corpora’s dictionary is not a
-
Page 928 and 929:
Finally, if we want to convert a li
-
Page 930 and 931:
Once we’re happy with the size an
-
Page 932 and 933:
from transformers import BertTokeni
-
Page 934 and 935:
"What about the separation token?"T
-
Page 936 and 937:
The last output, attention_mask, wo
-
Page 938 and 939:
Outputtensor([[ 3, 27, 1, ..., 0, 0
-
Page 940 and 941:
vector, right? And our vocabulary i
-
Page 942 and 943:
Maybe you filled this blank in with
-
Page 944 and 945:
Continuous Bag-of-Words (CBoW)In th
-
Page 946 and 947:
That’s a fairly simple model, rig
-
Page 948 and 949:
Figure 11.13 - Continuous bag-of-wo
-
Page 950 and 951:
Figure 11.15 - Reviewing restaurant
-
Page 952 and 953:
You got that right—arithmetic—r
-
Page 954 and 955:
There we go, 50 dimensions! It’s
-
Page 956 and 957:
Equation 11.1 - Embedding arithmeti
-
Page 958 and 959:
Only 82 out of 50,802 words in the
-
Page 960 and 961:
Now we can use its encode() method
-
Page 962 and 963:
Model I — GloVE + ClassifierData
-
Page 964 and 965:
Pre-trained PyTorch EmbeddingsThe e
-
Page 966 and 967:
Model Configuration & TrainingLet
-
Page 968 and 969:
6 self.encoder = encoder7 self.mlp
-
Page 970 and 971:
Figure 11.20 - Losses—Transformer
-
Page 972 and 973:
Outputtensor([[[2.6334e-01, 6.9912e
-
Page 974 and 975:
I want to introduce you to…ELMoBo
-
Page 976 and 977:
OutputToken: 32 watchThe get_token(
-
Page 978 and 979:
Helper Function to Retrieve Embeddi
-
Page 980 and 981:
Output(tensor(-0.5047, device='cuda
-
Page 982 and 983:
torch.all(new_flair_sentences[0].to
-
Page 984 and 985:
Outputtensor(0.3504, device='cuda:0
-
Page 986 and 987:
We can leverage this fact to slight
-
Page 988 and 989:
We can easily get the embeddings fo
-
Page 990 and 991:
Figure 11.24 - Losses—simple clas
-
Page 992 and 993:
We can inspect the pre-trained mode
-
Page 994 and 995:
Every word piece is prefixed with #
-
Page 996 and 997:
far, our models used these embeddin
-
Page 998 and 999:
position_ids = torch.arange(512).ex
-
Page 1000 and 1001:
Pre-training TasksMasked Language M
-
Page 1002 and 1003:
Then, let’s create an instance of
-
Page 1004 and 1005:
If these two sentences were the inp
-
Page 1006 and 1007:
The BERT model may take many other
-
Page 1008 and 1009:
The contextual word embeddings are
-
Page 1010 and 1011:
Model Configuration1 class BERTClas
-
Page 1012 and 1013:
"Which BERT is that? DistilBERT?!"D
-
Page 1014 and 1015:
Well, you probably don’t want to
-
Page 1016 and 1017:
set num_labels=1 as argument.If you
-
Page 1018 and 1019:
Output{'attention_mask': [1, 1, 1,
-
Page 1020 and 1021:
OutputTrainingArguments(output_dir=
-
Page 1022 and 1023:
Method for Computing Accuracy1 def
-
Page 1024 and 1025:
loaded_model = (AutoModelForSequenc
-
Page 1026 and 1027:
logits.logits.argmax(dim=1)Outputte
-
Page 1028 and 1029:
For a complete list of available ta
-
Page 1030 and 1031:
[215]. For a demo of GPT-2’s capa
-
Page 1032 and 1033:
in Chapter 9, and I reproduce it be
-
Page 1034 and 1035:
Data Preparation1 auto_tokenizer =
-
Page 1036 and 1037:
Data Preparation1 lm_train_dataset
-
Page 1038 and 1039:
The training arguments are roughly
-
Page 1040 and 1041:
device_index = (model.device.indexi
-
Page 1042 and 1043:
• learning that a language model
-
Page 1044 and 1045:
[167] https://huggingface.co/docs/d
Change language