Advanced Deep Learning with Keras

fourpersent2020
from fourpersent2020 More from this publisher
16.03.2021 Views

[ 229 ]Chapter 7titles = ('MNIST predicted source images.','SVHN predicted target images.','MNIST reconstructed source images.','SVHN reconstructed target images.')dirs = ('mnist_source-%s' % postfix, 'svhn_target-%s' % postfix)# genrate predicted target(svhn) and source(mnist) imagesif g_models is not None:g_source, g_target = g_modelsother_utils.test_generator((g_source, g_target),(test_source_data, test_target_data),step=0,titles=titles,dirs=dirs,show=True)return# build the cyclegan for mnist cross svhnmodels = build_cyclegan(shapes,"mnist-%s" % postfix,"svhn-%s" % postfix,kernel_size=kernel_size,patchgan=patchgan)# patch size is divided by 2^n since we downscaled the input# in the discriminator by 2^n (ie. we use strides=2 n times)patch = int(source_data.shape[1] / 2**4) if patchgan else 1params = (batch_size, train_steps, patch, model_name)test_params = (titles, dirs)# train the cyclegantrain_cyclegan(models,data,params,test_params,other_utils.test_generator)The results for transferring the MNIST from the test dataset to SVHN are shownin Figure 7.1.12. The generated images have the style of SVHN, but the digits arenot completely transferred. For example, on the 4 th row, digits 3, 1, and 3 are stylizedby CycleGAN. However, on the 3 rd row, digits 9, 6, and 6 are stylized as 0, 6, 01, 0, 65,and 68 for the CycleGAN without and with PatchGAN respectively.The results of the backward cycle are shown in Figure 7.1.13. In this case, the targetimages are from the SVHN test dataset. The generated images have the style ofMNIST, but the digits are not correctly translated. For example, on the 1 st row, thedigits 5, 2, and 210 are stylized as 7, 7, 8, 3, 3, and 1 for the CycleGAN without andwith PatchGAN respectively.

Cross-Domain GANsIn the case of PatchGAN, the output 1 is understandable given the predicted MNISTdigit is constrained to one digit. There are somehow correct predictions like in 2 ndrow last 3 columns of the SVHN digits, 6, 3, and 4 are converted to 6, 3, and 6 byCycleGAN without PatchGAN. However, the outputs on both flavors of CycleGANare consistently single digit and recognizable.The problem exhibited in the conversion from MNIST to SVHN where a digit inthe source domain is translated to another digit in the target domain is called labelflipping [8]. Although the predictions of CycleGAN are cycle-consistent, they arenot necessarily semantic consistent. The meaning of digits is lost during translation.To address this problem, Hoffman [8] introduced an improved CycleGAN calledCyCADA (Cycle-Consistent Adversarial Domain Adaptation). The difference is theadditional semantic loss term ensures that the prediction is not only cycle-consistentbut also sematic-consistent:Figure 7.1.12: Style transfer of test data from the MNIST domain to SVHN. Original color photo can be foundon the book GitHub repository, https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter7-cross-domain-gan/README.md.[ 230 ]

[ 229 ]

Chapter 7

titles = ('MNIST predicted source images.',

'SVHN predicted target images.',

'MNIST reconstructed source images.',

'SVHN reconstructed target images.')

dirs = ('mnist_source-%s' % postfix, 'svhn_target-%s' % postfix)

# genrate predicted target(svhn) and source(mnist) images

if g_models is not None:

g_source, g_target = g_models

other_utils.test_generator((g_source, g_target),

(test_source_data, test_

target_data),

step=0,

titles=titles,

dirs=dirs,

show=True)

return

# build the cyclegan for mnist cross svhn

models = build_cyclegan(shapes,

"mnist-%s" % postfix,

"svhn-%s" % postfix,

kernel_size=kernel_size,

patchgan=patchgan)

# patch size is divided by 2^n since we downscaled the input

# in the discriminator by 2^n (ie. we use strides=2 n times)

patch = int(source_data.shape[1] / 2**4) if patchgan else 1

params = (batch_size, train_steps, patch, model_name)

test_params = (titles, dirs)

# train the cyclegan

train_cyclegan(models,

data,

params,

test_params,

other_utils.test_generator)

The results for transferring the MNIST from the test dataset to SVHN are shown

in Figure 7.1.12. The generated images have the style of SVHN, but the digits are

not completely transferred. For example, on the 4 th row, digits 3, 1, and 3 are stylized

by CycleGAN. However, on the 3 rd row, digits 9, 6, and 6 are stylized as 0, 6, 01, 0, 65,

and 68 for the CycleGAN without and with PatchGAN respectively.

The results of the backward cycle are shown in Figure 7.1.13. In this case, the target

images are from the SVHN test dataset. The generated images have the style of

MNIST, but the digits are not correctly translated. For example, on the 1 st row, the

digits 5, 2, and 210 are stylized as 7, 7, 8, 3, 3, and 1 for the CycleGAN without and

with PatchGAN respectively.

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!