Config overhaul: one format, one file, structured schemas
- Dominant language
- Python
- Stars
- 21
- Forks
- 2
- Avg merge
- 6h 33m
- Merged PRs (30d)
- 91
Description
Tracking issue for the overhaul @dmarx asked for in #254 — unification and simplification of the whole config system, with omegaconf structured configs and YAML throughout.
## What is actually there today
```
luria.toml TOML the config
tags.yaml YAML per scheme, beside the records
statuses.yaml YAML per scheme, beside the records
.yaml YAML per scheme, declared by name (ADR-076)
remotes.lock.json JSON generated, committed
```
`config.py` is **2,263 lines**: 14 dataclasses, 19 module-level builders, and **44 `raise ValueError`**. It hand-parses a nested dict into typed objects, with `DEFAULTS` plus a bespoke `_merge`, plus `FAMILIES` for the replace-rather-than-merge rule from ADR-047. **35 modules import it.** `docs/configuration.md` is generated from the dataclass docstrings.
So: three formats, two locations, one hand-rolled loader.
## What omegaconf buys, concretely
- **The dataclasses already exist.** `Scheme`, `Reference`, `Vocabulary`, `Chain` are exactly what a structured config is. Schema, typing and defaults come free from what is already written.
- **`_merge` and `DEFAULTS` go away.** `OmegaConf.merge(schema, user)` is that function, and ADR-047's "a declared family replaces the default" becomes a merge policy rather than a special case in the loader.
- **The format split goes away.** One parser, one set of quoting rules.
- **Inheritance answers #254 directly.** `Reference` and `Vocabulary` share a base that carries `label` and `blurb`, so relations get described the same way vocabularies are — which was the original ask, and falls out of the type system instead of being bolted on.
## What it does not buy, and this is the part worth arguing about
**The 44 `ValueError`s are not type checks.** They are cross-field semantic rules: a converse must be mutual and point back at the declaring scheme; a derivation's `from` must name a reference that exists; `cite = "view"` requires `render = "document"`; `required_when` must name a field whose values can occur. Structured configs validate the *shape of a value*. None of those are about a value's shape.
So the simplification is real but bounded: it removes **plumbing**, not **checks**. I would expect the builders to shrink a lot and the validators to survive almost intact — and they should, because they are the part that earns its keep.
**And luria's error messages are better than omegaconf's.** This is a current one:
> `'extends' is not a reference LIT declares — a converse names the field holding the same relation read backwards, and it has to exist to be written into (declared: introduced_by)`
omegaconf would say something about a missing key of the wrong type. A migration that loses those messages makes the tool worse at the exact moment a user is confused. Whatever happens, the validation layer keeps speaking in luria's voice — structured configs go *underneath* it, not in place of it.
## Hazards I checked
- **No interpolation collision.** omegaconf resolves `${...}`; luria's templates are single-brace (`{n}`, `{1}.{2}`, `{uid}`). Nothing in this repo's config or the anthology's contains `${`. Safe.
- **Regex escaping is a real migration hazard.** `uid = "(\\d{4})[.:](\\d{4,5})"` in TOML does not carry over to YAML by copying the bytes — the quoting rules differ. This is exactly the kind of thing that converts silently and breaks at runtime, so the converter has to do it, not a person, and round-tripping every `uid`/`title_re` in a real record is the test.
- **Every existing record has a `luria.toml`.** Dual-read then deprecate; a format flip in one release is not an option.
- **`luria init` writes a heavily commented config**, built line by line in `init.py`, where the comments *are* the documentation. YAML keeps comments, so the idea survives — but that generator is a rewrite, not a port.
## Staged plan
The ordering is chosen so each stage is independently useful and independently revertible:
1. **#254's content model, no format change.** Give `Reference` a `label`/`blurb` sharing a base with `Vocabulary`. Unblocks the original ask now, with no migration and no new dependency.
2. **omegaconf as the validation layer behind the existing TOML parse.** Same input, same output, schema-checked underneath. No user-visible change — this is the stage that proves the dataclasses survive contact with structured configs before anything about the file format moves.
3. **Accept `luria.yaml`, dual-read.** TOML keeps working. `luria migrate config` converts, handling the regex-escaping hazard.
4. **Fold in the scattered vocabulary files** — see the open question below.
5. **Deprecate TOML at 1.0.**
## One open question, because I think it cuts against "one file"
> this doesn't need to live in multiple files
Agreed for `luria.toml` versus `luria.yaml` versus `remotes.lock.json`-as-JSON — that split is arbitrary.
I am less sure about `tags.yaml` and `statuses.yaml`. Those sit **beside the records they govern**, and that locality is doing work: a scheme's vocabulary is the one piece of config that a person editing documents in that directory actually reads, and `record/decisions.d/statuses.yaml` is discoverable from the directory you are already in. Centralizing them makes the config file complete at the cost of making the vocabulary remote from its use.
There is a real answer either way and it is your call — but it is a decision, not a cleanup, and it should get an ADR rather than arriving as a consequence of the format migration. A middle option exists: central by default, with a per-scheme `tags: path` override, which is roughly what `tags_file` already does.
Happy to take any stage. Stage 1 is small and unblocks #254 on its own, so unless you say otherwise I will start there.
Contributor guide
Research direction
Start with config.py and the staged plan, then read init.py and docs/configuration.md to understand current parsing and generated documentation. Begin with the independently useful Stage 1 tied to #254; completion should preserve existing TOML behavior while adding the shared Reference/Vocabulary content model, with later format migration treated as separate stages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100