pdfcoffee

Chapter 2
For multi-machine training, some machines are designated as workers and some
as parameter servers with the variables of the model placed on parameter servers.
Computation is replicated across all GPUs of all workers. Multiple workers can be
set up with the environment variable TF_CONFIG as in the following example:
os.environ["TF_CONFIG"] = json.dumps({
"cluster": {
"worker": ["host1:port", "host2:port", "host3:port"],
"ps": ["host4:port", "host5:port"]
},
"task": {"type": "worker", "index": 1}
})
In this section, we have seen how it is possible to train models using distributed
GPUs, multiple machines, and TPUs in a very simple way with very few additional
lines of code. Now let's see another difference between 1.x and 2.x. Namely,
namespaces.
Changes in namespaces
TensorFlow 2.x made a significant effort to clean the namespaces that in
TensorFlow 1.x became extremely dense, particularly in the root namespace,
making discoverability difficult. Here we have a summary of the major changes:
• tf.keras.layers: Contains all symbols that were previously under
tf.layers
• tf.keras.losses: Contains all symbols that were previously under
tf.losses
• tf.keras.metrics: Contains all symbols that were previously under
tf.metrics
• tf.debugging: A new namespace for debugging
• tf.dtypes: A new namespace for data types
• tf.io: A new namespace for I/O
• tf.quantization: A new namespace for quantization
TensorFlow 1.x currently provides over 2,000 endpoints in total, including over
500 endpoints in the root namespace. TensorFlow 2.x removed 214 endpoints,
including 171 endpoints in the root namespace. A conversion script has been added
to TensorFlow 2.x in order to facilitate the conversion from 1.x to 2.x and to highlight
deprecated endpoints.
[ 79 ]