Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Then, let’s use these layers to create functions that replicate both gates (r and z)
and the candidate hidden state (n):
def reset_gate(h, x):
thr = r_hidden(h)
txr = r_input(x)
r = torch.sigmoid(thr + txr)
return r # red
def update_gate(h, x):
thz = z_hidden(h)
txz = z_input(x)
z = torch.sigmoid(thz + txz)
return z # blue
def candidate_n(h, x, r):
thn = n_hidden(h)
txn = n_input(x)
n = torch.tanh(r * thn + txn)
return n # black
Cool—all the transformations and activations are handled by the functions above.
This means we can replicate the mechanics of a GRU cell at its component level (r, z,
and n). We also need an initial hidden state and the first data point (corner) of a
sequence:
initial_hidden = torch.zeros(1, hidden_dim)
X = torch.as_tensor(points[0]).float()
first_corner = X[0:1]
We use both values to get the output from the reset gate (r):
r = reset_gate(initial_hidden, first_corner)
r
Output
tensor([[0.2387, 0.6928]], grad_fn=<SigmoidBackward>)
632 | Chapter 8: Sequences