pdfcoffee

Chapter 2
In our toy example we use NumPy for generating training data x and labels
y, and we transform it into tf.data.Dataset with tf.data.Dataset.from_
tensor_slices(). Then we apply a shuffle to avoid bias in training across
GPUs and then generate SIZE_BATCHES batches:
import tensorflow as tf
import numpy as np
from tensorflow import keras
N_TRAIN_EXAMPLES = 1024*1024
N_FEATURES = 10
SIZE_BATCHES = 256
# 10 random floats in the half-open interval [0.0, 1.0).
x = np.random.random((N_TRAIN_EXAMPLES, N_FEATURES))
y = np.random.randint(2, size=(N_TRAIN_EXAMPLES, 1))
x = tf.dtypes.cast(x, tf.float32)
print (x)
dataset = tf.data.Dataset.from_tensor_slices((x, y))
dataset = dataset.shuffle(buffer_size=N_TRAIN_EXAMPLES).
batch(SIZE_BATCHES)
2. In order to distribute some computations to GPUs, we instantiate a
distribution = tf.distribute.MirroredStrategy() object, which
supports synchronous distributed training on multiple GPUs on one
machine. Then, we move the creation and compilation of the Keras model
inside the strategy.scope(). Note that each variable in the model is
mirrored across all the replicas. Let's see it in our toy example:
# this is the distribution strategy
distribution = tf.distribute.MirroredStrategy()
# this piece of code is distributed to multiple GPUs
with distribution.scope():
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(16, activation='relu', input_
shape=(N_FEATURES,)))
model.add(tf.keras.layers.Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.SGD(0.2)
model.compile(loss='binary_crossentropy', optimizer=optimizer)
model.summary()
# Optimize in the usual way but in reality you are using GPUs.
model.fit(dataset, epochs=5, steps_per_epoch=10)
[ 77 ]