Clarification on patch query initialization in local encoder

Hi team, thank you for your great work!

I'm having trouble understanding how the patch queries are initialized in the local encoder. The paper states:

> where $\mathbf{P} \in \mathbb{R}^{n_p \times d_{fg}}$ represents $n_p$ patch representations to be processed by the global model, which is initialized by pooling together the byte embeddings $\mathbf{e}_i$ corresponding to each patch $p_i$.

However, in the code, it seems these representations are pooled from the output of one of the local encoder layers. Here's the relevant call stack:

- https://github.com/facebookresearch/blt/blob/2dcf48bdd96bf7d3d868e5d799534eda8774b7fa/bytelatent/model/local_models.py#L275
- https://github.com/facebookresearch/blt/blob/2dcf48bdd96bf7d3d868e5d799534eda8774b7fa/bytelatent/model/local_models.py#L281
- https://github.com/facebookresearch/blt/blob/2dcf48bdd96bf7d3d868e5d799534eda8774b7fa/bytelatent/model/local_models.py#L293

From what I understand:
- If `self.cross_attn_all_layers_encoder == True`, the patch queries are pooled from $h_1$
- Otherwise, they're pooled from $h_{n-1}$

Have I misunderstood this behavior, or is there a mismatch between the paper and the code? If so, which one reflects the intended design?

Start in bytelatent/model/local_models.py at lines 275, 281, and 293, then trace how patch queries are pooled under both cross_attn_all_layers_encoder settings. Compare that behavior with the paper’s initialization description. Done means the intended design is established and the paper or code is clarified to remove the discrepancy.

Written by the indexing model from the issue text.