ContextLab / ContextLab/orchestrator

Example pipelines: 50 of 117 validate; the remaining 67 classified

Open
#104 3 comments 0 reactions 0 assignees View on GitHub
bug documentation enhancement high-priority
Dominant language
Python
Stars
3
Forks
2
Avg merge
13m
Merged PRs (30d)
1

Description

## Current state (measured 2026-08-03)

**50 of 117 example pipelines pass `orchestrator validate`.**

The original title said "44 of 45 fail". Both numbers are stale: the catalogue
has grown, and most of the movement since has come from fixing *the product*
rather than the examples.

| | count |
|-|-|
| validating | 50 |
| not validating | 67 |
| total | 117 |

Note that 25 of the 67 are `enhanced/` twins of files already counted, so the
number of *distinct* pipelines is about 83. Fixing a twin fixes both.

## Classification of the 67

Tracked by class rather than by a single number, so that progress is legible
and a regression in supported examples is not hidden by churn elsewhere.

| class | count | meaning |
|-|-|-|
| duplicate/archive | 25 | `enhanced/` twin of a file already in the list |
| unsupported: tool missing | 11 | references a tool the product does not provide |
| repairable: undefined refs | 10 | genuine authoring errors, fixable in the file |
| legacy: schema | 8 | pre-dates the current pipeline schema |
| legacy: malformed | 8 | does not parse |
| legacy: stale refs | 4 | references steps/outputs that no longer exist |
| other | 1 | |

`output_path` alone accounts for 26 references across the "undefined refs"
class and is genuinely undeclared — it needs a product decision, not an edit.

## Product defects found by triaging rather than repairing

Classifying the failures by their actual `validate` output — instead of
editing the files — turned up defects where the *validator* was wrong and the
example was fine. Anyone repairing these by hand would have "fixed" correct
files:

1. A JSON Schema was read as a model specification; the `TypeError` escaped
and was reported as "Model validation failed", i.e. as though the pipeline
were invalid. (#457)
2. `execution['timestamp']` was looked up as a task id while
`execution.timestamp` — the same reference — validated. (#457)
3. `json_encode` did not exist. (#457)
4. References were extracted by chopping text, producing names nobody wrote
(`'(row'`, `'from_json)'`, `'analysis_topics[loop'`) and reporting them
against the pipeline. (#458)
5. `{{ timestamp }}` and `{{ execution.timestamp }}` were different clocks —
four hours and 29 microseconds apart in one run. (#459)
6. Every read of a run's context started a new run; execution ids collided
within a second; two engines offered a `pipeline` namespace the validators
refuse. (#460)
7. Reference extraction suppressed a name everywhere if it was bound
anywhere, so a typo outside a loop disappeared when a loop elsewhere in the
file used the same name. (#461)

## Archived examples

The 13 examples listed in the original description were removed in `5c2dacb`
("Remove example files and associated tests"). They are recoverable from git
history:

```
git show 5c2dacb^:examples/.yaml
```

The `/tmp/orchestrator_examples_backup.tar.gz` referenced in the original
description **no longer exists** and should not be relied on. Nothing outside
the repository is durable tracking.

## Remaining work

- [ ] Repair the 10 genuine authoring errors
- [ ] Decide whether `output_path` is a declared pipeline input or an authoring error
- [ ] Decide the fate of the 25 `enhanced/` twins (not deletion — they are kept deliberately)
- [ ] Migrate or retire the 20 legacy-syntax files
- [ ] Decide whether the 11 missing tools are to be built or the examples retired
- [ ] CI dashboard reporting counts by class, failing on supported-example
regressions rather than on the headline number

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.