NVIDIA / NVIDIA/TensorRT-LLM

[Bug][AutoDeploy]: model-coverage entries fail because config rejects use_cache

Open
#14,672 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

AutoDeploy AutoDeploy/Dashboard bug Customized kernels
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • CPU architecture: x86_64
  • GPU properties: not captured from CI artifact
  • Libraries
    • TensorRT-LLM branch or tag: main (pipeline context)
    • TensorRT-LLM commit: not captured here
    • Container used: AutoDeploy model-coverage CI container
  • OS: CI environment
  • Additional context: terminal pipeline 52785950, job 327395444
Who can help?

No response

Information
  • The official example scripts
  • My own modified scripts
Tasks
  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)
Reproduction

Pipeline 52785720 resolved to terminal model-coverage pipeline 52785950.

Affected registry entries in examples/auto_deploy/model_registry/models.yaml:

  • apple/OpenELM-270M-Instruct
  • apple/OpenELM-1_1B-Instruct
  • apple/OpenELM-3B-Instruct

Representative evidence:

  1. Open job 327395444.
  2. Inspect per-test stdout logs for the three OpenELM tests.
  3. Each traceback ends with:
TypeError: OpenELMConfig.post_init() got an unexpected keyword argument 'use_cache'

Relevant stack tail:

File "/opt/tensorrt-llm/tensorrt_llm/_torch/pyexecutor/config_utils.py", line 479, in load_pretrained_config
  model_config = transformers.AutoConfig.from_pretrained(
...
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 190, in init
  self.post_init(**additional_kwargs)
TypeError: OpenELMConfig.post_init() got an unexpected keyword argument 'use_cache'
Expected behavior

OpenELM model-coverage entries should load HF config successfully through the current AutoDeploy/llmapi config-loading path.

actual behavior

TensorRT-LLM fails during HF config loading before model execution starts because OpenELMConfig.post_init() rejects the use_cache keyword argument.

additional notes

This reproduces consistently for all three OpenELM instruct entries. The current local registry has those entries disabled with this reason. Likely fix areas are the TensorRT-LLM config-loading path or OpenELM integration path so unsupported kwargs are filtered or normalized before config instantiation.

Before submitting a new issue...
  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tensorrt_llm/_torch/pyexecutor/config_utils.py, especially load_pretrained_config, and inspect the three OpenELM entries in examples/auto_deploy/model_registry/models.yaml. Use pipeline 52785950 and job 327395444 logs to reproduce the config-loading failure. Done means all three OpenELM model-coverage entries load their HF configuration successfully through the current AutoDeploy path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.