[RFC] Context-aware defaults for better OOTB LLMAPI experience

Problem Statement

• Currently in TensorRT-LLM, the default arguments for LLMAPI are set globally, to provide the best-effort default experience. They make sure "most" of the models work OOTB, with light tuning. They don't make any guarantees on performance.
• Users must still manually configure model-specific settings to avoid crashes and achieve optimal performance as a blanket default list cannot cover compatibility/optimality across all models and all scenarios.
• The users are pointed to the model support documentation to figure out the support-matrix for the model they are interested in, individual model-specific quick-start guides (like this one for Gemma3 VL that requires explicitly setting the attention backend, kv cache reuse and chunked prefill params to make it work)

Current Pain Points

1. Manual Configuration Required
   • Config files specify enable_block_reuse: false (e.g., examples/configs/curated/qwen3-next.yaml) to get the model functional.
   • Chunked prefill (for long context support) isn't enabled by default (vllm, sglang provides it by default) - because some models like VLMs/DeepSeekv3.2 don't support it.
   • The model support page documents certain edge-cases like the following:
     • Chunked Prefill for MLA can only be enabled on SM100/SM103.
     • KV cache reuse for MLA can only be enabled on SM90/SM100/SM103 and in BF16/FP8 KV cache dtype.
     • Overlap scheduler isn't supported when using EAGLE-3(Two Model Engine) for GPT-OSS.

Therefore the user needs to be aware of this and manually configure in such situations. Although we provide in-depth deployment guides, we don't provide intelligence in the default case.

2. Missed Performance Opportunities
   • DeepSeek models could benefit from automatic MTP (Multi-Token Prediction) configuration for specific scenarios.
   • Models miss hardware-specific optimizations.
   • Better CUDA graph batching configurations that can be enabled automatically and encoded programmatically with context at the time of instantiation.

Proposed Implementation

• The core issue is that, to set the best-case defaults, we need all the context we can about what model is used, what hardware, what are the user-provided LLM Args, defaults of neighboring params in LLM API to programmatically encode best defaults.

• We need a model-aware and context-aware defaults resolution system using clean two-phase loading approach.

• Fortunately, BaseCheckpointLoader already has separate methods for config and weights. The checkpoint loader can call load_config() without loading large weight tensors into memory - useful for config inspection and planning before committing to expensive weight load. The model files are already cached on disk from the download phase (HuggingFace models are downloaded once and cached locally), so there's no additional network I/O.

• Since LLM Args are Pydantic models, we can leverage Pydantic's model_fields_set attribute which tracks which fields were explicitly set by the user vs using defaults. This allows us to precisely identify unset fields where we can apply model-specific defaults without overriding user intentions:

  # Pydantic tracks which fields were explicitly set
  llm_args = TorchLlmArgs(max_batch_size=32)  # User set this
  llm_args.model_fields_set  # {'max_batch_size'}
  
  # We only apply defaults to fields NOT in model_fields_set
  if 'enable_chunked_prefill' not in llm_args.model_fields_set:
      # Safe to apply model-specific default
      llm_args.enable_chunked_prefill = model_specific_value
  

• The decision-making of which defaults to set can be informed from the rich context of the default + user-provided config we have parsed already.

Architecture Design

graph TB
    A[User Request] --> B[Phase 1: Load Config Only]
    B --> C[Detect Model Type]
    C --> D[Apply Dynamic Defaults: model-specific, context-aware]
    D --> H[Apply User Overrides]
    H --> E[Phase 2: Create Executor and Instantiate Model]
    E --> F[Load Full Model]

    style D fill:#90EE90

Key Components of Proposed Implementation

1. Model-Specific Defaults Method

class DeepSeekV3ForCausalLM(DecoderModelForCausalLM):
    @classmethod
    def get_model_defaults(cls, llm_args: TorchLlmArgs) -> dict:
        """Return model-specific defaults based on context."""
        defaults = {}

        # Auto-enable MTP for low-latency scenarios
        if llm_args.max_batch_size <= 16 and not llm_args.speculative_config:
            defaults["speculative_config"] = {
                "algorithm": "mtp_eagle",
                "num_nextn_predict_layers": 3
            }

        # Hardware-aware settings
        if detect_gpu() in ["H100", "H200"]:
            defaults["attention_backend"] = "fp8_flashMLA"

        # Memory optimization for long context
        if llm_args.max_seq_len > 64000:
            defaults["kv_cache_config"] = {"free_gpu_memory_fraction": 0.6}

        return defaults

2. Two-Phase Loading Implementation

   The key innovation is introducing a lightweight config-only loading phase that applies model defaults BEFORE creating runtime features:

   # Phase 1: Load config and apply model-specific defaults
   checkpoint_loader = _construct_checkpoint_loader(...)
   llm_args = ModelLoader.load_config_and_apply_defaults(
       checkpoint_dir, llm_args, checkpoint_loader
   )
   # At this point, llm_args has model-specific defaults applied
   
   # Phase 2: Create runtime features with correct settings
   enable_chunked_context = llm_args.enable_chunked_prefill  # Has model-specific value
   attn_runtime_features = AttentionRuntimeFeatures(
       chunked_prefill=enable_chunked_context,  # e.g., False for VLMs
       cache_reuse=kv_cache_config.enable_block_reuse,  # e.g., False for Qwen3Next
       ...
   )
   # Critical: These settings must be set BEFORE model instantiation
   
   # Phase 3: Load actual model with everything configured correctly
   model_engine = PyTorchModelEngine(...)
   

   Why this timing matters: AttentionRuntimeFeatures are used for construction-time decisions that cannot be changed after model loading. Without two-phase loading, VLMs crash with chunked prefill enabled.

3. User Override Preservation

   • Model defaults never override explicit user settings
   • Clean precedence: User settings > Model defaults > Global defaults

Expected Benefits

This architectural tweak is powerful because it gives the ability to configure a dynamic optimal default that is model-specific and context-specific to each model developer - and directly translates to an improved OOTB experience to the user.

1. Just works OOTB

• When certain models don't support certain defaults OOTB, the model-specific defaults takes care of it. (for eg. Qwen3Next can auto-disable enable_block_reuse within itself without affecting other places)
• Models that support long context will do so by default.

2. Performance Improvements

• Automatic MTP enablement for compatible scenarios
• Hardware-aware attention backend selection
• Optimal CUDA graph batch size configuration
• Context-aware memory management etc.

3. Code Simplification and Maintenance

• Model-specific tuning for best defaults are co-located with the model code.
• Model devs with best knowledge of perf can express it in code instead of documentation.