lablup / lablup/backend.ai

[ModelDeploy] Destroy the endpoint when deployment create fails to add its first revision

Open
#13,392 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Goal

A deployment create that fails while building its first revision must leave no endpoint row behind.

## Background

`DeploymentService.create_deployment` (`services/deployment/service.py:449-474`):

```python
deployment_info = await self._deployment_controller.create_deployment(action.creator) # commit #1
if action.creator.model_revision is not None:
await self._deployment_controller.add_deployment_revision(...) # commit #2..N
```

Step 1 commits the endpoint as `PENDING` with no revision. Step 2 opens its own sessions and can raise from at least seven places:

| Raise site | Condition |
|---|---|
| `db_source.py:2593` | requester cannot reach the model vfolder via `query_accessible_vfolders` |
| `deployment_controller.py:408` / `:413` | preset repository unavailable, or neither request nor preset supplies `runtime_variant_id` |
| `deployment_controller.py:428` / `:430` | merge chain produced no `image_id` / no `mounts` |
| `deployment_controller.py:436` | `ModelDefinitionDraft.to_resolved()` rejects a definition with no `models[].name` |
| `deployment_controller.py:466` | `RequiredResourceSlotRule` finds a required slot missing or non-positive |
| `reader.py:170-184` | malformed `deployment-config.yaml` / `model-definition.yaml` on the model vfolder |
| `deployment_controller.py:568` | `activate_revision` fails after the revision row was already written |

The orphan is permanent: `EndpointLifecycle.PENDING` appears in no handler's `target_statuses()`, so the coordinator never sees the row. The caller received a GraphQL error and the WebUI never navigated, so nobody associates the leftover row with the failed click.

`create_legacy_deployment` (`services/deployment/service.py:476-506`) has the identical two-step shape and needs the same treatment.

## Acceptance Criteria

- When `add_deployment_revision` raises, the endpoint created moments earlier is destroyed and the original error is re-raised unchanged — callers keep seeing the real cause, not a cleanup error.
- A cleanup that itself fails is logged and does not mask the original exception.
- The same holds for `create_legacy_deployment`.
- Unit coverage: `create_deployment` with a revision step that raises leaves no endpoint; the raised error is the revision error.
- The stale `check_pending` reference in `deploying_initializing.py:30` is removed.

## Notes

The narrower fix is a compensating delete in the service method — one guard covering every caller (model-card deploy, vfolder deploy, deployment create v2, legacy create). Extending the endpoint-creation transaction to span the revision write is the stronger repair but reaches into the repository layer and the RBAC creator; if we take that route it should be scoped separately.

`createModelDeploymentV2` with `initial_revision: null` must keep working — it never enters the branch above.

JIRA Issue: BA-XXXX

Contributor guide

Open the contributing guide

Research direction

Start in services/deployment/service.py:449-506 and trace create_deployment and create_legacy_deployment through add_deployment_revision. Review the listed failure sites and deploying_initializing.py:30, then add focused unit coverage for a revision failure. Done means the endpoint is removed, the original error is preserved, cleanup failures are logged without masking it, and the stale check_pending reference is gone.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.