pdfcoffee

Advanced Convolutional Neural Networks
"Atrous" is the "bastardization" of the French expression "à trous,"
meaning "with holes." So an AtrousConvolution is a convolution
with holes.
In short, in D-dilated convolution, usually the stride is 1, but nothing prevents
you from using other strides. An example is given in the following diagram with
increased dilatation (hole) sizes = 0, 1, 2:
Thanks to this simple idea of introducing "holes," it is possible to stack multiple
dilated convolutional layers with exponentially increasing filters, and learn longrange
input dependencies without having an excessively deep network.
A WaveNet is therefore a ConvNet where the convolutional layers have various
dilation factors, allowing the receptive field to grow exponentially with depth
and therefore efficiently cover thousands of audio timesteps.
When we train, the inputs are sounds recorded from human speakers. The
waveforms are quantized to a fixed integer range. A WaveNet defines an initial
convolutional layer accessing only the current and previous input. Then, there is
a stack of dilated convnet layers, still accessing only current and previous inputs.
At the end, there is a series of dense layers combining the previous results followed
by a softmax activation function for categorical outputs.
At each step, a value is predicted from the network and fed back into the input.
At the same time, a new prediction for the next step is computed. The loss function
is the cross entropy between the output for the current step and the input at the
next step. The following image shows the visualization of a WaveNet stack and its
receptive field as introduced by Aaron van den Oord [9]. Note that generation can be
slow because the waveform has to be synthesized in a sequential fashion, as x t
must
be sampled first in order to obtain x >t
where x is the input:
[ 180 ]