Rust-style source-located diagnostics for TOML config validation errors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 99
- Forks
- 62
- Avg merge
- 6d 12h
- Merged PRs (30d)
- 17
Description
Summary
When a test/scenario TOML fails Pydantic validation, the user gets a flattened message with no source location — no file line, no pointer to the offending key. For cross-field errors (e.g. the env_params ⇄ cmd_args contract added in #901) the user has to manually grep the file to find which key and which annotation are in conflict. We can do much better with rustc/miette-style diagnostics that quote the exact lines and point carets at the conflict.
Motivation (current UX)
Take a config where cmd_args.osl is a scalar but is annotated [env_params.osl] (rejected at parse time since #901). Today:
ERROR Failed to parse test spec: 'conf/staging/.../dse_ppo.toml'
Validation error: Value error, env_params['osl'] annotates cmd_args.osl, which is not a candidate list (got int); the annotation only reclassifies a list-valued sweep as env-sampled, while a scalar is already fixed. Make cmd_args.osl a list or remove the annotation
There is no line number. In a 200-line config with several [env_params.*] blocks, the user still has to hunt for both the osl = 512 line and the [env_params.osl] block.
Proposed enhancement
Render a Rust-style, source-located diagnostic that pinpoints both lines in conflict:
error: env_params['osl'] annotates cmd_args.osl, which is not a candidate list (got int)
--> conf/staging/.../dse_ppo.toml:63
|
38 | osl = 512
| ^^^ cmd_args.osl defined here
...
63 | [env_params.osl]
| ^^^^^^^^^^^^^^^^^ env_params['osl'] annotation declared here
|
= help: the annotation only reclassifies a list-valued sweep as env-sampled, while a scalar is already fixed. Make cmd_args.osl a list or remove the annotation
The value: the user sees the file:line, the two conflicting lines quoted, carets on the exact spans, and the fix in a help: footer — no grepping.
Design notes
- Positions don't survive parsing.
toml.loadreturns a plain dict, so line/col are lost. The fix is to re-read the source text on the error path (the parser already holdsself.current_fileinTestParser.load_test_definition) and locate the keys by scanning, then render. - Keep validators source-agnostic. Pydantic validators (e.g.
TestDefinition.validate_env_params) raise the structured error; the source-located rendering lives in the parse/format layer (toml_utils+test_parser). Construction without a file (DSE overlay, programmatic use) still gets today's plain message. - Structured errors over string-matching. Raising
PydanticCustomError("env_params_scalar_target", msg, {"key": name})lets the formatter key offerr["type"]/ctxinstead of regex-matching the message text (more robust than sniffing the human string). - Graceful fallback. When a span can't be located (e.g. inline-table
cmd_args = { osl = 512 }forms), fall back to today's plain message — never crash the error path. - Generalizable.
env_paramsis the first beneficiary, but the same machinery (map a Pydanticerr["loc"]path → TOML key span → caret) applies to any field validation error, plus we already emit located messages for TOML syntax errors (format_toml_decode_error). This unifies both under one renderer.
Scope / effort
Small and additive (~150–200 LoC): a TOML key/section locator + a multi-span caret renderer in toml_utils.py, wired into TestParser.load_test_definition, behind the existing ValidationError path. No behavior change to successful parses.
Severity
Enhancement (DX/UX). Pure quality-of-life; highest leverage for cross-field config contracts like the env_params annotations introduced in #901.
Code: src/cloudai/test_parser.py (load_test_definition), src/cloudai/toml_utils.py.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/cloudai/test_parser.py, especially load_test_definition, and src/cloudai/toml_utils.py, including format_toml_decode_error; trace the existing ValidationError path and the structured env_params validation. Add source-located multi-span diagnostics with a graceful plain-message fallback, while keeping validators source-agnostic and successful parses unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100