Feedback about NLP From Scratch: Generating Names with a Character-Level RNN

There is the following issue on this page: https://docs.pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html.

There is no non-linearity wrapper while passing the hidden state onto the next iteration, and on output layer that predicts distribution. Results are coming relatively fine though, maybe because nn.LogSoftmax is implicitly acting as non-linearity. But for learning hidden states, its bad, $h_t$ is literally linear combination of $h_{t-1}$, all past $x$ s and $Cat$.

Simple Fix:

class RNN(nn.Module):
    def __init__(self,C_in:int, C_hid:int, C_out:int):
        super().__init__()
        self.C_in = C_in
        self.C_hid = C_hid
        self.C_out = C_out
        self.C_cat = data1.C_cat
        self.i2h = nn.Linear(data1.C_cat + C_in + C_hid,C_hid)
        self.i2o1 = nn.Linear(data1.C_cat + C_in + C_hid,C_out)
        self.o12o2 = nn.Linear(C_out + C_hid,C_hid)
        self.o22o3 = nn.Linear(C_hid, C_out)
        self.tanh = nn.Tanh()
        self.todist = nn.LogSoftmax(dim=1)
        
    def forward(self,x_C:tensor, x_X:tensor, x_H:tensor):
        # Inputs = (B=1,C_cat), (B=1, C_in), (B=1,C_out), 
        # Output = (B=1, C_out), (B=1,C_hid)
        x = torch.cat((x_C,x_X,x_H),dim=1)
        h = self.tanh(self.i2h(x))
        x = self.i2o1(x)
        x = torch.cat((x,h),dim=1)
        x = self.tanh(self.o12o2(x))
        x = self.o22o3(x)

        return self.todist(x), h
        

Edit1: nn.tanh 's range is between $[-1,1]$. Compressing logits before softmax can really impact the result. Using one more layer before projection gave me similar error of 2.2127 , and nice samples.

Thank you.

Start with the linked NLP From Scratch tutorial page and inspect the character-level RNN implementation and its training example. Reproduce the reported behavior, evaluate whether the proposed activation changes are correct, and update the tutorial only if the issue is confirmed while preserving working training and generated samples.