ContextLab / ContextLab/clustrix

Restore the Lambda Cloud backend: removed from v0.2.0 as never verified end to end

Open
#146 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
4
Avg merge
6h 27m
Merged PRs (30d)
9

Description

The Lambda Cloud VM backend (`cluster_type="lambda_cloud"`) is implemented and removed from v0.2.0 as unverified. No Lambda Cloud job has been shown to run end to end.

## What exists, and where

Introduced in `6430b7d` (2025-06-28). At `299109f`:

| Code | Lines |
|-|-|
| `clustrix/cloud_providers/lambda_cloud.py` | 495 |
| `clustrix/cost_providers/lambda_cloud.py` | 322 |
| `clustrix/pricing_clients/lambda_pricing.py` | 330 |
| `clustrix/kubernetes/lambda_provisioner.py` | 686 (see the Kubernetes issue) |

Shared with the other three providers: `clustrix/executor_cloud.py`, `clustrix/cloud_providers/base.py`, `clustrix/pricing_clients/base.py`, and the `lambda_*`/`cloud_*` configuration fields. Tutorial: `docs/source/notebooks/lambda_cloud_tutorial.ipynb`.

## Why this one deserves particular care on restoration

Lambda Cloud rents GPU instances by the hour and has no free tier. A backend that provisions an instance and then fails to reach it -- which is exactly what the placeholder-hostname behaviour did -- leaves a billing instance running with no way for clustrix to find it again. Any restoration should start with the teardown path, not the submission path.

## Why it is being removed rather than fixed

Not because the code is known to be wrong. Because it has **never been run against the real thing**, and shipping it in the cluster-type dropdown states otherwise. A user who selects it gets a code path no one has ever seen succeed.

v0.2.0 keeps exactly the four backends that have been demonstrated end to end -- `local`, `ssh`, `slurm`, `huggingface` -- and the documentation now says the rest are planned for a future release rather than currently supported.

## Restoring it

Nothing is lost: every line cited above stays reachable in git history at the commits named. Reinstating it means reverting the removal commit and then doing the part that was never done -- running it against real hardware and recording the evidence in this issue.

## Definition of done

- [ ] Backend restored from history
- [ ] A real job submitted, executed and its result returned, with the transcript pasted into this issue
- [ ] Failure paths exercised (job rejected, job killed, node lost)
- [ ] Re-added to `SUPPORTED_CLUSTER_TYPES`, the widget dropdown and the CLI
- [ ] Documentation moved from "planned" to "supported"

Contributor guide

Open the contributing guide

Research direction

Start with the removal commit and the historical implementations in `clustrix/cloud_providers/lambda_cloud.py`, `clustrix/cost_providers/lambda_cloud.py`, and `clustrix/pricing_clients/lambda_pricing.py`; read the teardown path first because failed connectivity can leave a billable instance running. Then run a real Lambda Cloud job and exercise the listed failure paths. Done means restoring the backend, documenting the end-to-end transcript here, re-adding it to the supported-type, widget and CLI lists, and updating the documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.