ContextLab / ContextLab/clustrix

Restore the AWS execution backend: removed from v0.2.0 as never verified end to end

Open
#143 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
4
Avg merge
6h 27m
Merged PRs (30d)
9

Description

The AWS execution backend (`cluster_type="aws"`) is implemented and removed from v0.2.0 as unverified. No AWS job has been shown to run end to end.

## What exists, and where

Introduced in `31d3b38` (2025-06-28). At `299109f`:

| Code | Lines |
|-|-|
| `clustrix/cloud_providers/aws.py` | 893 |
| `clustrix/cost_providers/aws.py` | 369 |
| `clustrix/pricing_clients/aws_pricing.py` | 273 |
| `clustrix/kubernetes/aws_provisioner.py` | 872 (see the Kubernetes issue) |

Shared with the other three providers: `clustrix/executor_cloud.py` (585 lines, cloud VM dispatch), `clustrix/cloud_providers/base.py`, `clustrix/pricing_clients/base.py`, and the `aws_*`/`cloud_*` configuration fields.

## Defects found while auditing

- **Cloud results were deserialized with no signature and no key.** The signed-payload protection added for SSH and SLURM was not wired into the cloud path, so a compromised instance could return an arbitrary pickle. Fixed in the v0.2.0 sweep, never exercised against real AWS.
- The provider returned **placeholder hostnames** (`placeholder.example.com`) and empty strings as though they were real instances, so failures surfaced far from their cause.

## Related work that is NOT removed

`scripts/aws/` (resource cleanup and cluster destruction) stays. It is operator tooling rather than an execution backend, it defaults to a dry run, and it refuses to touch anything not tagged `clustrix:managed=true`. See #95.

Also see #87, which records unresolved EKS access problems (a Service Control Policy blocking cluster creation) encountered the last time anyone tried to run this for real.

## Why it is being removed rather than fixed

Not because the code is known to be wrong. Because it has **never been run against the real thing**, and shipping it in the cluster-type dropdown states otherwise. A user who selects it gets a code path no one has ever seen succeed.

v0.2.0 keeps exactly the four backends that have been demonstrated end to end -- `local`, `ssh`, `slurm`, `huggingface` -- and the documentation now says the rest are planned for a future release rather than currently supported.

## Restoring it

Nothing is lost: every line cited above stays reachable in git history at the commits named. Reinstating it means reverting the removal commit and then doing the part that was never done -- running it against real hardware and recording the evidence in this issue.

## Definition of done

- [ ] Backend restored from history
- [ ] A real job submitted, executed and its result returned, with the transcript pasted into this issue
- [ ] Failure paths exercised (job rejected, job killed, node lost)
- [ ] Re-added to `SUPPORTED_CLUSTER_TYPES`, the widget dropdown and the CLI
- [ ] Documentation moved from "planned" to "supported"

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.