ContextLab / ContextLab/orchestrator
Example pipelines: 50 of 117 validate; the remaining 67 classified
- Dominant language
- Python
- Stars
- 3
- Forks
- 2
- Avg merge
- 13m
- Merged PRs (30d)
- 1
Description
## Current state (measured 2026-08-03)
**50 of 117 example pipelines pass `orchestrator validate`.**
The original title said "44 of 45 fail". Both numbers are stale: the catalogue
has grown, and most of the movement since has come from fixing *the product*
rather than the examples.
| | count |
|-|-|
| validating | 50 |
| not validating | 67 |
| total | 117 |
Note that 25 of the 67 are `enhanced/` twins of files already counted, so the
number of *distinct* pipelines is about 83. Fixing a twin fixes both.
## Classification of the 67
Tracked by class rather than by a single number, so that progress is legible
and a regression in supported examples is not hidden by churn elsewhere.
| class | count | meaning |
|-|-|-|
| duplicate/archive | 25 | `enhanced/` twin of a file already in the list |
| unsupported: tool missing | 11 | references a tool the product does not provide |
| repairable: undefined refs | 10 | genuine authoring errors, fixable in the file |
| legacy: schema | 8 | pre-dates the current pipeline schema |
| legacy: malformed | 8 | does not parse |
| legacy: stale refs | 4 | references steps/outputs that no longer exist |
| other | 1 | |
`output_path` alone accounts for 26 references across the "undefined refs"
class and is genuinely undeclared — it needs a product decision, not an edit.
## Product defects found by triaging rather than repairing
Classifying the failures by their actual `validate` output — instead of
editing the files — turned up defects where the *validator* was wrong and the
example was fine. Anyone repairing these by hand would have "fixed" correct
files:
1. A JSON Schema was read as a model specification; the `TypeError` escaped
and was reported as "Model validation failed", i.e. as though the pipeline
were invalid. (#457)
2. `execution['timestamp']` was looked up as a task id while
`execution.timestamp` — the same reference — validated. (#457)
3. `json_encode` did not exist. (#457)
4. References were extracted by chopping text, producing names nobody wrote
(`'(row'`, `'from_json)'`, `'analysis_topics[loop'`) and reporting them
against the pipeline. (#458)
5. `{{ timestamp }}` and `{{ execution.timestamp }}` were different clocks —
four hours and 29 microseconds apart in one run. (#459)
6. Every read of a run's context started a new run; execution ids collided
within a second; two engines offered a `pipeline` namespace the validators
refuse. (#460)
7. Reference extraction suppressed a name everywhere if it was bound
anywhere, so a typo outside a loop disappeared when a loop elsewhere in the
file used the same name. (#461)
## Archived examples
The 13 examples listed in the original description were removed in `5c2dacb`
("Remove example files and associated tests"). They are recoverable from git
history:
```
git show 5c2dacb^:examples/.yaml
```
The `/tmp/orchestrator_examples_backup.tar.gz` referenced in the original
description **no longer exists** and should not be relied on. Nothing outside
the repository is durable tracking.
## Remaining work
- [ ] Repair the 10 genuine authoring errors
- [ ] Decide whether `output_path` is a declared pipeline input or an authoring error
- [ ] Decide the fate of the 25 `enhanced/` twins (not deletion — they are kept deliberately)
- [ ] Migrate or retire the 20 legacy-syntax files
- [ ] Decide whether the 11 missing tools are to be built or the examples retired
- [ ] CI dashboard reporting counts by class, failing on supported-example
regressions rather than on the headline number
Contributor guide
Assessment
This issue has not been assessed yet.