pdfcoffee

Chapter 5
Then the two feature vectors (one for the image, and one for the text) are combined
into one joint vector that is provided as input to a dense network to produce the
combined network:
# combine the encoded question and visual model
merged = layers.concatenate([encoded_question, visual_model])
# attach a dense network at the end
output = layers.Dense(1000, activation='softmax')(merged)
# get the combined model
vqa_model = models.Model(inputs=[image_input, question_input],
outputs=output)
vqa_model.summary()
For instance, if we have a set of labeled images, then we can learn what the best
questions and answers are for describing an image. The number of options is
enormous! If you want to know more, I suggest that you investigate Maluuba,
a start-up providing the FigureQA dataset with 100,000 figure images and
1,327,368 question-answer pairs in the training set. Maluuba has been recently
acquired by Microsoft, and the lab is advised by Yoshua Bengio, one of the fathers
of deep learning.
In this section, we have discussed how to implement VQA. The next section is about
style transfer: a deep learning technique used for training neural networks to create
art.
Style transfer
Style transfer is a funny neural network application that provides many insights
into the power of neural networks. So what exactly is it? Imagine that you observe
a painting made by a famous artist. In principle you are observing two elements:
the painting itself (say, the face of a woman, or a landscape) and something more
intrinsic, the "style" of the artist. What is the style? That is more difficult to define,
but humans know that Picasso had his own style, Matisse had his own style, and
each artist has his/her own style. Now, imagine taking a famous painting of Matisse,
giving it to a neural network, and letting the neural network repaint it in Picasso's
style. Or imagine taking your own photo, giving it to a neural network, and having
your photo painted in Matisse's or Picasso's style, or in the style of any other artist
that you like. That's what style transfer does.
[ 165 ]