pdfcoffee

Tensor Processing Unit
Then the usage of TPUs, where the appropriate TPU strategy is used:
with strategy.scope():
model = create_model()
model.compile(optimizer=tf.keras.optimizers.SGD(nesterov=True,
momentum=0.9),
loss='categorical_crossentropy',
metrics=['accuracy'])
model.summary()
In short, it is extremely simple to use TPUs with the upcoming TensorFlow 2.1, and
if you want to experiment immediately you can use TensorFlow 2.0 nightly build.
Martin reports typical run times for his specific model:
• GPU (V100): 15s per epoch
• TPU v3-8 (8 cores): 5s per epoch
• TPU pod v2-32 (32 cores): 2s per epoch
Summary
TPUs are very special ASIC chips developed at Google for executing neural
network mathematical operations in an ultra-fast manner. The core of the
computation is a systolic multiplier that computes multiple dot products (row
* column) in parallel, thus accelerating the computation of basic deep learning
operations. Think of a TPU as a special-purpose coprocessor for deep learning, which
is focused on matrix or tensor operations. Google has announced three generations
of TPUs so far, plus an additional Edge TPU for IoT. Cloud TPU v1 is a PCI-based
specialized co-processor, with 92 TeraFLOPS and inference only. Cloud TPU v2
achieves 180 TeraFLOPS and it supports training and inference. Cloud TPU v2 pods
released in alpha in 2018 can achieve 11.5 PetaFLOPS. Cloud TPU v3 achieves 420
TeraFLOPS with both training and inference support. Cloud TPU v3 pods can deliver
more than 100 PetaFLOPS of computing power. That's a world-class supercomputer
for tensor operations!
[ 588 ]