lablup / lablup/backend.ai

Deployment create from the model store can leave a revision-less, permanently stuck endpoint

Open
#13,391 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Goal

Make "Deploy" from the model store either produce a deployment that actually rolls out, or produce nothing at all. Today it can produce a third thing: an endpoint row with no revision, stuck in `PENDING` forever.

## Background

`DeploymentService.create_deployment` (`services/deployment/service.py:449`) creates a deployment in two independently committed steps:

1. `DeploymentController.create_deployment` writes the endpoint with `revision=None` (`sokovan/deployment/deployment_controller.py:216`) and `lifecycle_stage=PENDING` (`repositories/deployment/creators/deployment.py:163`). Its session commits at `repositories/deployment/db_source/db_source.py:319`.
2. `DeploymentController.add_deployment_revision` (`deployment_controller.py:506`) resolves the revision from the preset chain, persists it, and calls `activate_revision`, which is the only writer that sets `deploying_revision_id` and moves the endpoint to `DEPLOYING` (`db_source.py:3025`).

There is no transaction spanning the two steps and no compensating delete. Any failure in step 2 leaves a durable endpoint with `current_revision_id = NULL`, `deploying_revision_id = NULL`, `lifecycle_stage = PENDING`. No lifecycle handler targets `PENDING`, so the coordinator never advances, retries, or reaps it — the row sits in the deployment list forever while the caller only saw a GraphQL error.

The model-store Deploy path (`deployModelCardV2` → `api/adapters/model_card/adapter.py:470`) is the most exposed caller because the WebUI sends only a preset id and a resource group. Image, resource slots, runtime variant, cluster config and model definition are all resolved *inside* step 2 from preset and runtime-variant data the caller never validated, so every "required X" check in `add_revision` (`deployment_controller.py:376-481`) fires after the endpoint is already committed.

Two of those checks can fail deterministically for data that the write APIs currently accept:

- a revision preset with no required resource slot rows (`sokovan/deployment/validators/required_resource_slot_rule.py:71`)
- a runtime variant whose `default_model_definition` yields no `models[].name` — the column is `NOT NULL` but `{}` is a legal value (`models/runtime_variant/row.py:33`), so it passes the DB check and fails at `to_resolved()` (`deployment_controller.py:436`)

## Acceptance Criteria

- A failed deploy leaves no endpoint row behind, on both the v2 and the legacy create paths.
- A revision preset or runtime variant that cannot produce a deployable revision is rejected when it is written, not when someone deploys with it.
- The preset-derived deployment-level fallbacks cannot produce a null replica count or an unbound strategy spec.
- Deploying from the model store with a valid preset still succeeds end to end, verified against a live server.

## Out of scope

- `createModelDeploymentV2` with `initial_revision: null` (`api/adapters/deployment/adapter.py:687`) intentionally creates a revision-less deployment for the two-step create → `addModelRevision` → `activateRevision` flow. `PENDING` with no revision stays a supported state for that mutation; only the *failed* create path must clean up after itself.
- Reaping endpoints already orphaned in existing deployments — an operational cleanup, not a code change.

## Sub-issues

- [ ] Deployment create leaves a revision-less endpoint when the revision step fails
- [ ] Revision presets and runtime variants accept values that always fail at deploy time
- [ ] Preset-derived deployment-level resolution has unguarded fallbacks

## Related

- `deploying_initializing.py:30` still documents a `check_pending` step that no longer exists anywhere in the codebase.
- Split introduced in a86e17357 (#11167), which moved revision creation out of the endpoint-creation transaction.

JIRA Issue: BA-XXXX

Contributor guide

Open the contributing guide

Research direction

Trace services/deployment/service.py:449 through sokovan/deployment/deployment_controller.py and repositories/deployment/db_source/db_source.py, comparing the v2 and legacy create paths. Review the validator and runtime-variant files named in the issue first; done means failed creates leave no endpoint, invalid inputs are rejected earlier, and a valid model-store deployment succeeds against a live server.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.