Nimblesite / Nimblesite/Basilisk
Unparseable files: text output drops them entirely, and a user syntax error exits 3 (Internal failure) masking exit 1
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 54
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
A file Basilisk cannot parse is handled correctly in JSON and badly everywhere
else. Three separate defects share one cause: CheckOutcome::failures is a
side-channel that only the JSON renderer knows about.
[CHKARCH-CLI-OUTPUT-FAILURES] already argues this case for JSON, and its
reasoning applies verbatim to text:
Omitting it rendered
[]— byte-for-byte the answer a clean file gets — for a
file that was never checked, so every consumer reading the report rather than
the exit code was told a file with a syntax error had no problems. […] a
report that contradicts it is worse than one that says nothing.
The spec section is implemented and tested only in
basilisk-cli/src/output/json.rs.
Text has no counterpart, no spec clause, and no test.
Repro
$ mkdir /tmp/bsk_syn && cd /tmp/bsk_syn
$ printf 'x: int = "no"\n' > good.py
$ printf 'obj = object()\nobj.\n' > broken.py
1. Text output drops the unanalysable file from the report and the count
$ basilisk check .
error[assignment_compatibility]: Type mismatch: `x` is annotated `int` (int) but assigned LiteralString
--> ./good.py:1:1
...
Found 1 diagnostic (1 error).
2026-08-01T06:52:51Z ERROR basilisk: error processing file path=./broken.py error=syntax error in ./broken.py: Expected an identifier at byte range 20..21
The report body has no entry for broken.py, and the summary says 1
diagnostic. --output json on the same run returns 2 entries. Text and JSON
disagree about what the run found.
The only signal in text mode is a tracing line on stderr, emitted after the
summary and outside the report — so it is lost to 2>/dev/null, to any log
filter, and to anyone reading the report rather than watching the console. With
no other diagnostics present, the text renderer prints nothing at all:
// crates/basilisk-cli/src/main.rs:426
let exit_code = if total == 0 && outcome.failures.is_empty() {
println!("{}", "All checked. No issues found.".green().bold());
0
} else if total == 0 {
0 // <- failures present: prints nothing, reports nothing
}
That branch was clearly added to stop the "All checked. No issues found." lie —
correct instinct, but silence is the other half of the same bug.
2. A user syntax error exits 3 (Internal failure) and masks exit 1
$ basilisk check . # 1 type error + 1 unparseable file
$ echo $?
3
$ rm broken.py && basilisk check . ; echo $?
1
[CHKARCH-CLI-EXITCODES] defines 3 as Internal failure. A syntax error in
the user's own Python is not internal to Basilisk — it is the most ordinary
finding a checker can have. Two consequences:
- CI that branches on the documented codes (
[CHKARCH-CLI-CI]tells consumers to
do exactly that) is told the tool broke, not that the code has problems.
Pipelines that retry or page on3will do so on a plain typo. - The failure exit overrides the diagnostic exit unconditionally
(if outcome.failures.is_empty() { diagnostic_exit } else { 3 },
main.rs:360), so one
unparseable file hides the fact that real errors were found.
This reproduces on this repository: basilisk check . exits 3 because
basilisk-zed/tests/fixtures/completions.py ends in obj. — a deliberately
incomplete completion fixture. The fixture is correct and must stay invalid; the
handling of it is what is wrong.
3. The position is a byte range, not line:col
syntax error in ./broken.py: Expected an identifier at byte range 20..21
No user can act on byte 20. Every other checker reports file:2:5: invalid syntax. The offset comes straight from ruff_python_parser, which hands us a
TextRange — the line/column conversion the rest of the pipeline already does
for diagnostics is simply never applied here. The entry also carries no code and
therefore no see: URL.
Note the JSON entry anchoring at line 1, column 1 is specified behaviour and
should not change; this is about the message text.
Acceptance
- Extend
[CHKARCH-CLI-OUTPUT-FAILURES]to cover text output, and render
unanalysable files in the text report with the same standing they have in
JSON — visible in the body, counted in the summary, on stdout with the
report rather than as a stderr log line. - Decide and specify the exit code for an unanalysable file.
1matches its
severity and keeps3meaning what the table says. If it must stay
distinct from1, it needs its own row in[CHKARCH-CLI-EXITCODES]— not
a reuse of "Internal failure". - Stop a failure from masking a real diagnostic exit.
- Convert the parser's byte offset to
line:colin the message. - Regression tests for each: a text-mode assertion mirroring the existing
cli_binary_tests.rsJSON assertions, a text-vs-JSON agreement test on one
mixed run, and an exit-code test for failure-only and failure-plus-error
trees.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in crates/basilisk-cli/src/main.rs around lines 360 and 426, then compare the existing failure handling in crates/basilisk-cli/src/output/json.rs. Read the related cli_binary_tests.rs JSON assertions and the invalid basilisk-zed/tests/fixtures/completions.py fixture. Done means text and JSON agree on failure entries and counts, parser positions are line:col, and failure-only and mixed runs use specified exit codes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- cli, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100