ContextLab / ContextLab/clustrix

Red-team campaign: trust boundary, script injection, serialization fuzzing, cost safety

Open
#126 1 comment 0 reactions 0 assignees View on GitHub
P2-medium red-team security
Dominant language
Python
Stars
10
Forks
4
Avg merge
6h 27m
Merged PRs (30d)
9

Description

Part of #108 · **Phase 6** · label: red-team · **Do this after Phases 0-5, not before.**

Red-teaming a codebase whose test suite cannot run and whose cloud path has never executed would just rediscover known defects. This issue is for adversarial review of the *hardened* system.

## Scope

### 1. The remote -> local trust boundary
The deepest structural risk (see the Phase 3 security issue). Once payload authentication lands, attack it:
- [ ] Tampered `result.pkl` with a valid-looking structure -> must be rejected
- [ ] Replayed payload from a previous job -> must be rejected
- [ ] Malicious `error.pkl` (the error path is easy to forget to authenticate)
- [ ] A job that writes an oversized/zip-bomb result -> must fail gracefully, not OOM the client
- [ ] Confirm the restricted unpickler cannot be escaped via `__reduce__`

### 2. Injection into generated remote scripts
Script generation concatenates strings (`utils.py:1149-1186`, `executor_kubernetes.py:152-243`, `executor_cloud.py:376-414`), and `utils.py:1123` emits unquoted `export {var}={value}`.
- [ ] Hostile values in every interpolated config field: `memory`, `time`, `partition`, `queue`, env var names/values, `remote_work_dir`, `cluster_name`
- [ ] Shell metacharacters, newlines, `$(...)`, backticks, null bytes, unicode lookalikes
- [ ] Same for the K8s payload's nested quoting around its base64 blob

### 3. Serialization / function-capture fuzzing
- [ ] Functions with closures over mutable state, generators, coroutines, lambdas, nested decorators, C extensions, unpicklable members
- [ ] Deeply recursive and cyclic argument structures
- [ ] Functions defined in a REPL (the known limitation — assert a *clear* error, per the docs issue)
- [ ] Confirm `utils.py:81`'s `eval(range_part)` replacement cannot be driven to execute arbitrary code
- [ ] Confirm `executor_kubernetes.py:217`'s `exec(cleaned_source, ...)` reconstruction is safe or gone

### 4. Credential handling
- [ ] Confirm no credential reaches argv, `/proc//environ` in a way that surprises, a log, a traceback, or a temp file that outlives the process
- [ ] Verify file modes under a hostile umask
- [ ] Confirm the GCP temp-file fix actually unlinks on crash, not just on clean exit

### 5. Concurrency and resource exhaustion
- [ ] Many simultaneous submissions -> file-descriptor and thread growth (ties to the resource-leak issue)
- [ ] Job-ID collisions; the busy-poll dict at `executor_cloud.py:438` under concurrent mutation
- [ ] Cancellation and timeout paths — currently several waits have no timeout at all

### 6. Cost safety
- [ ] Attempt to make the test suite spend money; it must be impossible without explicit opt-in (regression test for #109)
- [ ] Verify provisioning failures always tear down; no orphaned billable resources

## Method

- [ ] Property-based testing (Hypothesis) for serialization and script generation
- [ ] An independent reviewer who did not write the fixes — the audit that produced this plan found that self-review consistently missed the *seams*
- [ ] `bandit` / `pip-audit` / CodeQL wired into CI
- [ ] Every finding gets a regression test before it is closed

## Acceptance criteria

- [ ] Documented red-team report with reproduction steps for each finding
- [ ] Every finding either fixed with a regression test, or accepted with written rationale
- [ ] Security tooling runs in CI and blocks on new high-severity findings

Contributor guide

Open the contributing guide

Research direction

Start only after Phases 0-5, then review the trust boundary and generated-script paths named in utils.py:81, 1123, 1149-1186, executor_kubernetes.py:152-243 and 217, and executor_cloud.py:376-414, 438. Use adversarial and property-based tests to exercise serialization, injection, credentials, concurrency and cost safety. Done means a documented report with reproductions, regression tests or written rationale for every finding, and security tooling running in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, github-actions, kubernetes, python
Domain
cloud, distributed-systems, security, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.