Architectural clarifications, decoder's halting condition

Hello and thanks for all the work done so far, I find the BLT architecture very inspiring and I am looking forward to integrate it in new experiments. For this reason, I am currently working on a new implementation from scratch, with the goal of being as compact and clean as possible.

Unfortunately, there are some logic passages that I do not find very clear neither in the paper nor in your implementation.

Let's revise the logic (ignoring on purpose hash embedding):
1) A sequence is passed to the BLT, for example "[BOS]Hello, world![EOS]" (seq_len=16)
2) The entropy model decides the patch boundaries, for example: {'[BOS]',H,e,l},{'l','o'},{','},{'w','o','r'},{'l','d','!',[EOS]} (note: this is random, just to give the concept).
3) In this case, 5 patches are created by pooling the patch's bytes embedding together (ex. patches[3]= Pool('w','o','r'))
3) The local encoder starts with a vector h of seq_len bytes embeddings, here we have an alternation of "bytes-embedding" layers, employing a classical causal + sliding window attention mask, and cross attention layers, where each patch can only attend to its bytes. So patch[3] will attend to bytes[7:9]. We have only a "document" kind of mask (patch attend to all its bytes) but no longer a causal or sliding mask here.
4) The local encoder will output two tensors: the vector h of bytes embeddings, transformed by the layers, in this case of length 16, and a vector of patches, 5 in this example;
5) The global transformer will receive 5 patches as input and returns 5 patches as output;
6) The local decoder will receive both the patches as well as the original h vector coming from the encoder. Here we are going to have a causal+sliding mask on self-attention (performed over h vector) and again the same "document" mask in the cross-attention so that each byte in h is allowed only to attend to its patch.

Now, I really don't know if this understanding is correct. I am able to replicate it in simple code, but I have doubts:

1) So the local decoder, apart from patches, will receive the original byte embedding vector h and it's going to output byte embeddings. I am bit worried by this point because I wonder if in this way the decoder could learn to focus too much on the original bytes embeddings and less on patches (where the big computation happen). Perhaps I am just overthinking this: the model will learn where it should seek information into patches and where this information is so easy that can be retrieved just from the embedding. Still, I am worried that in this way the patches could lose important information. Won't it be possible to just input patches to the decoder? Have you tried this config (clearly, then another way shall be found to signal sequence length)? What is your justification for including the byte embeddings, passed through the encoder, into the decoder?

2) I have an "halting problem" issue. So, during training, the goal is to minimize the loss on a sequence of fixed length => the decoder is going to receive, still referring to the example, 5 patches and 16 bytes' embeddings. Fine: it will return 16 bytes and the loss can be computed. But, during inference, how does it knows where its work of decoding bytes from a given patch ends? The only way in which I see this working is that during inference it will just work byte-by-byte, that is, the length of h and of the paches vector will be the same, 1, until EOS is decoded. But in this case, it would mean that during inference all the richness of the architecture get lost and the model collapse on a charformer. There is clearly something I do not understand here.

Any hint? I would really like to completely understand this architecture, but I find the paper lacking some subtle details and the code just too complex because I can see that it includes a lot of experiments that were done for the paper, but this doesn't help to understand what is the core of the architecture. I think these clarification could be helpful to other people too, so that your architecture can get implemented, cited and get all the attention it deserves.

Thank you!

Compare the paper's decoder description with the implementation; start by tracing the local decoder's inputs and inference flow, since no specific files are named. Document whether byte embeddings are passed through and how decoding determines its stopping point; done means both questions are answered consistently for training and inference.

Written by the indexing model from the issue text.