[Feature] Improve multi-layer decoder of custom backends

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Most of the custom backends of SpecForge forward their multiple decoder layers with shared caches (including cache_position and past_key_values). It is possible to create separate caches for each decoder layer for better cache utilization.

Related resources

There isn't too much documentation/benchmarks around cache_position. The name seems to suggest it is a cache used by a decoder layer.

past_key_values, however, are usually typed as containers of tensors of dimension num_hidden_layers/n_layers on huggingface across models.

• past_key_values of GPT2
• mem of Transformer XL
• past_key_values of LayoutLM

Start by locating the custom backend decoder implementations and tracing how cache_position and past_key_values are forwarded across decoder layers. Compare their current behavior with the linked Hugging Face GPT2, Transformer XL, and LayoutLM cache descriptions; done means defining and validating the separate-cache behavior, including its cache-utilization impact.