Improve error handling during deployment draft resolve to surface root cause
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Problem
When POST /services (and the v2 deployment-revision flow) builds the merged RevisionDraft and calls to_model_revision_spec → ModelDefinitionDraft.to_resolved → ModelConfigDraft.to_resolved, failures surface as opaque errors that do not tell the operator which merge layer is at fault.
Concrete examples observed in BAI 26.4.4rc3:
1. The vfolder model-definition.yaml fails Pydantic validation (e.g. legacy string start_command, hyphenated start-command key, unknown field). The exception is raised inside fetch_model_definition, then swallowed by RevisionDraftReader._read_vfolder_drafts with a generic warning ("Failed to read model-definition.yaml from vfolder <id>, skipping"). The merge silently drops the yaml layer.
2. The merged ModelConfigDraft ends up with name=None because every layer that could have supplied it was either missing (custom variant has default_model_definition.models=null), gated off (variant.reads_vfolder_config_files=False), or silently skipped (case 1). ModelConfigDraft.to_resolved finally raises bare ValueError("ModelConfig.name is required"), which the API maps to HTTP 500. The operator has no way to know that the actual cause was a yaml validation error logged minutes earlier.
Goal
Make draft resolve failures self-explanatory. An operator looking at a single error response or log line should be able to tell:
- Which merge layer (variant baseline, preset, deployment-config.yaml, model-definition.yaml, request) contributed (or failed to contribute) to the field that ended up missing or invalid.
- For yaml/preset parse failures, the original validation error including field path and offending value, not a generic "skipping" warning.
- For final resolve failures, which fields are missing and which sources were tried.
Scope
Applies to the legacy and v2 deployment create / add-revision flows that go through:
- src/ai/backend/manager/sokovan/deployment/revision_draft/reader.py (RevisionDraftReader._read_vfolder_drafts and friends)
- src/ai/backend/manager/repositories/deployment/storage_source/storage_source.py (fetch_model_definition, fetch_deployment_config)
- src/ai/backend/manager/sokovan/deployment/deployment_controller.py (build_creator_from_legacy_draft / equivalent v2 path)
- src/ai/backend/common/config.py (ModelDefinitionDraft.to_resolved, ModelConfigDraft.to_resolved)
Suggested direction (open for discussion)
- Promote vfolder yaml parse failures from "log warning + skip" to a domain exception (subclass of BackendAIError) that carries vfolder id, file path, and the underlying ValidationError details. Decide whether to surface this as 4xx to the caller (preferred when the user owns the yaml) or keep it as a structured warning when the merge can still complete.
- When ModelConfigDraft.to_resolved (or analogous resolvers) raise for a missing required field, attach context: the per-field provenance map of which draft layers had a value and which were None, so the error message reads like "ModelConfig.name is required (variant baseline: empty, preset: empty, deployment-config.yaml: not read, model-definition.yaml: parse failed — see InvalidAPIParameters above, request: empty)".
- Add structured logging tags (vfolder_id, layer, field) so operators can grep one request id across the merge pipeline.
Acceptance
- A custom variant POST /services request with a malformed model-definition.yaml returns a 4xx (or 5xx with explicit cause) whose message identifies the yaml file and the failing field — not "ModelConfig.name is required".
- A request whose yaml is missing entirely returns an error that lists the layers that were tried and explains which field could not be resolved.
- Existing successful create flows (vllm/nim/sglang/etc., custom with valid yaml) are unchanged.
- Unit tests cover at least: malformed yaml, missing yaml for custom variant, well-formed yaml with missing model name field.
Related
- BA-5965 (legacy string start_command coercion) — fixes one of the trigger conditions but does not address the diagnosability gap.
- See attached analysis "ModelConfig.name 누락 — custom variant 서비스 생성 실패" for the pipeline trace and reproduction.
JIRA Issue: BA-5966
Contributor guide
Research direction
Start by tracing the legacy and v2 deployment create flows through src/ai/backend/manager/sokovan/deployment/revision_draft/reader.py and the storage source methods, then inspect the resolver methods in src/ai/backend/common/config.py. Compare how malformed, missing, and incomplete YAML drafts are handled and how deployment_controller.py builds each flow. Done means errors identify the failing layer, field, source, and underlying validation details, with unit coverage for the three acceptance cases and successful flows unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100