Context windows: a model is never told where its window sits on the timeline
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
Context windowing produces a picture that flickers at the window period on
MiniMax-H3, from the second window onward. The cause is general rather than
H3-specific, which is why this is an issue and not a pull request.
## What happens
`IndexListContextHandler` slices the latent per window and calls the model, but
nothing in the window's state reaches the model. `get_resized_cond` receives the
`IndexListContextWindow` and resizes conditioning against it, and the window's
`index_list` is never passed down to the forward pass.
For a model that positions its target from a cursor, every window is therefore
positioned identically. In `comfy/ldm/minimax/model.py`, `PackedLayout.__init__`:
```python
cursor = float(text_len)
for blk in refs or ():
cursor += _ref_t_span(blk)
...
segments.append(("video", n_video))
pos.append(_video_grid(latent_t, frame, cursor))
```
`cursor` is `text_len` plus the reference spans and does not depend on the
window. Every window's target rows are told they begin at the clip origin, so
each renders the opening of the shot, and the overlaps then crossfade between
two openings.
Confirmed at runtime by logging the target's start position per layout build:
167 builds across a 9-window run, one distinct value.
## Repro
Any H3 render through `ContextWindowsManual` with a clip long enough for two or
more windows. Log `cursor` in `PackedLayout.__init__` and it is constant. The
picture is clean through window 0 and flickers thereafter.
## Why it is general
Nothing here is unique to H3 — it is simply a model where the symptom is
visible, because it packs text, references and target into one sequence with
explicit time coordinates. Any model that derives target positions from a
per-call origin has the same gap. The information exists in the handler and is
discarded.
A local fix that works: give the layout a `window_start` in pixel frames and
offset the target grids by it, using the same unit and rate the model's own
keyframe anchors use. `window_start=0` reproduces current behaviour exactly.
With it applied, a 192-frame 3-window render goes from flickering to smooth.
The upstream shape is presumably a documented way for the handler to pass a
window's position into `extra_conds` / the forward pass, so a model can use it
if the concept means something to it. Happy to prepare that if the approach is
agreed.
## Second, related finding
`causal_window_fix` (default True) prepends one anchor frame to every window
after the first. On a model whose VAE groups frames non-uniformly this puts
those windows off the model's own frame grid: H3's grouping is (1,4,4,4,4), so
27 latent frames is exactly 90 pixel frames while 28 is 94, which is not a legal
run for it at all. Window 0 is 27 and every other window is 28.
That is likely correct for the causal-VAE models it was written for, but it has
no way to know whether a model's temporal grouping is uniform. Worth a hook, or
at least a note that it assumes one latent frame is one time step.
## Version
ComfyUI 0.34.0, PyTorch 2.10.0+cu130, Windows, RTX 4090.
Contributor guide
Research direction
Start with IndexListContextHandler and get_resized_cond to trace how IndexListContextWindow state reaches the forward pass, then inspect PackedLayout.__init__ in comfy/ldm/minimax/model.py. Determine the agreed upstream path for carrying the window position into model layout construction. Done means later context windows receive distinct timeline positions while window 0 retains current behavior and the multi-window H3 render no longer flickers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100