eellak / eellak/glossAPI

Docs: document full artifact contract per pipeline stage, aligned with smoke-test expectations

Open
#92 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
141
Forks
41
PR merge metrics
No merged PRs in 30d

Description

### Artifact layout in pipeline docs doesn't fully match what the smoke test expects

While setting up the repo locally and reading through the pipeline docs alongside
the smoke test, I noticed some artifacts that `test_pipeline_smoke.py` checks for
aren't really surfaced in `docs/pipeline.md`.

The docs do a good job covering the main outputs of each stage, but a few
runtime and debug artifacts that show up during actual runs are easy to miss
as a new contributor:

- `clean_markdown/` — produced by `Corpus.clean()` separately from `markdown/`
- `.processing_state.pkl` — the resume/state file persisted across runs
- `problematic_files/` and `timeout_files/` — triage outputs from cleaning

I ran into this because the contribution notes say that when outputs move or
change, `docs/pipeline.md` and `tests/test_pipeline_smoke.py` should be updated
together. That made me want to check whether the two were actually in sync,
and they're not quite there yet.

### What I'd propose

I'd like to update `docs/pipeline.md` to reflect the full per-stage artifact
layout, matching what the smoke test asserts. I think it would also help to
distinguish between:

- **Canonical outputs** — the intended deliverables of each stage
- **Runtime/resume artifacts** — things like `.processing_state.pkl` that
support crash recovery
- **Debug/triage artifacts** — folders like `problematic_files/` that help
diagnose failures

This way, the next contributor who modifies a stage output has a clear
reference for what should exist, what the smoke test will check, and what
needs to be updated together.

Happy to hear if this taxonomy makes sense or if you'd prefer a different
way to organise it before I draft the PR.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.