Deployment create from the model store can leave a revision-less, permanently stuck endpoint
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Goal
Make "Deploy" from the model store either produce a deployment that actually rolls out, or produce nothing at all. Today it can produce a third thing: an endpoint row with no revision, stuck in `PENDING` forever.
## Background
`DeploymentService.create_deployment` (`services/deployment/service.py:449`) creates a deployment in two independently committed steps:
1. `DeploymentController.create_deployment` writes the endpoint with `revision=None` (`sokovan/deployment/deployment_controller.py:216`) and `lifecycle_stage=PENDING` (`repositories/deployment/creators/deployment.py:163`). Its session commits at `repositories/deployment/db_source/db_source.py:319`.
2. `DeploymentController.add_deployment_revision` (`deployment_controller.py:506`) resolves the revision from the preset chain, persists it, and calls `activate_revision`, which is the only writer that sets `deploying_revision_id` and moves the endpoint to `DEPLOYING` (`db_source.py:3025`).
There is no transaction spanning the two steps and no compensating delete. Any failure in step 2 leaves a durable endpoint with `current_revision_id = NULL`, `deploying_revision_id = NULL`, `lifecycle_stage = PENDING`. No lifecycle handler targets `PENDING`, so the coordinator never advances, retries, or reaps it — the row sits in the deployment list forever while the caller only saw a GraphQL error.
The model-store Deploy path (`deployModelCardV2` → `api/adapters/model_card/adapter.py:470`) is the most exposed caller because the WebUI sends only a preset id and a resource group. Image, resource slots, runtime variant, cluster config and model definition are all resolved *inside* step 2 from preset and runtime-variant data the caller never validated, so every "required X" check in `add_revision` (`deployment_controller.py:376-481`) fires after the endpoint is already committed.
Two of those checks can fail deterministically for data that the write APIs currently accept:
- a revision preset with no required resource slot rows (`sokovan/deployment/validators/required_resource_slot_rule.py:71`)
- a runtime variant whose `default_model_definition` yields no `models[].name` — the column is `NOT NULL` but `{}` is a legal value (`models/runtime_variant/row.py:33`), so it passes the DB check and fails at `to_resolved()` (`deployment_controller.py:436`)
## Acceptance Criteria
- A failed deploy leaves no endpoint row behind, on both the v2 and the legacy create paths.
- A revision preset or runtime variant that cannot produce a deployable revision is rejected when it is written, not when someone deploys with it.
- The preset-derived deployment-level fallbacks cannot produce a null replica count or an unbound strategy spec.
- Deploying from the model store with a valid preset still succeeds end to end, verified against a live server.
## Out of scope
- `createModelDeploymentV2` with `initial_revision: null` (`api/adapters/deployment/adapter.py:687`) intentionally creates a revision-less deployment for the two-step create → `addModelRevision` → `activateRevision` flow. `PENDING` with no revision stays a supported state for that mutation; only the *failed* create path must clean up after itself.
- Reaping endpoints already orphaned in existing deployments — an operational cleanup, not a code change.
## Sub-issues
- [ ] Deployment create leaves a revision-less endpoint when the revision step fails
- [ ] Revision presets and runtime variants accept values that always fail at deploy time
- [ ] Preset-derived deployment-level resolution has unguarded fallbacks
## Related
- `deploying_initializing.py:30` still documents a `check_pending` step that no longer exists anywhere in the codebase.
- Split introduced in a86e17357 (#11167), which moved revision creation out of the endpoint-creation transaction.
JIRA Issue: BA-XXXX
Contributor guide
Research direction
Trace services/deployment/service.py:449 through sokovan/deployment/deployment_controller.py and repositories/deployment/db_source/db_source.py, comparing the v2 and legacy create paths. Review the validator and runtime-variant files named in the issue first; done means failed creates leave no endpoint, invalid inputs are rejected earlier, and a valid model-store deployment succeeds against a live server.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100