[Usage]: Question about the progress of "Eagle3 static draft tree"

System Info

System Information:

• OS: centos
• TensorRT-LLM version: tensorrt-llm 1.2.0rc6

How would you like to use TensorRT-LLM

I am currently experimenting with the Eagle3 algorithm, specifically focusing on the static draft token tree functionality.

During my testing, I observed a counter-intuitive behavior regarding the eagle_choices configuration:

• Configuration A: eagle_choices = [[0], [0, 0]]
• Configuration B: eagle_choices = [[0], [1], [0, 0]]

Observation: Configuration B results in a shorter accept length than Configuration A, which is unlikely to happen based on my understanding.

Investigation & Suspicions

After further investigation, I suspect the KV cache rewinding logic may not yet be fully ready for the static draft token tree structure. It seems that trtllm maintains a integer req.py_rewind_len in sampler.py to do KV cache rewinding, but a single integer is not sufficient to rewind from a tree-like structure I think.

I noticed that PR #8586 mentioned a bug related to KV cache rewinding. I would like to clarify:

1. Has the bug mentioned in #8586 been fully resolved in the current main branch?
2. Does my observation align with known limitations of the current KV cache management for Eagle3?

I want to know more about the progress of the "eagle3 static draft tree function", thank you if you can provide more information or give me ref docs.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.