Nimblesite / Nimblesite/Basilisk

Unparseable files: text output drops them entirely, and a user syntax error exits 3 (Internal failure) masking exit 1

Open
#384 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

high-priority spec-violation
Dominant language
Rust
Stars
54
Forks
3
PR merge metrics
No merged PRs in 30d

Description

A file Basilisk cannot parse is handled correctly in JSON and badly everywhere
else. Three separate defects share one cause: CheckOutcome::failures is a
side-channel that only the JSON renderer knows about.

[CHKARCH-CLI-OUTPUT-FAILURES] already argues this case for JSON, and its
reasoning applies verbatim to text:

Omitting it rendered [] — byte-for-byte the answer a clean file gets — for a
file that was never checked, so every consumer reading the report rather than
the exit code was told a file with a syntax error had no problems. […] a
report that contradicts it is worse than one that says nothing.

The spec section is implemented and tested only in
basilisk-cli/src/output/json.rs.
Text has no counterpart, no spec clause, and no test.

Repro

$ mkdir /tmp/bsk_syn && cd /tmp/bsk_syn
$ printf 'x: int = "no"\n' > good.py
$ printf 'obj = object()\nobj.\n' > broken.py
1. Text output drops the unanalysable file from the report and the count
$ basilisk check .
error[assignment_compatibility]: Type mismatch: `x` is annotated `int` (int) but assigned LiteralString
 --> ./good.py:1:1
...
Found 1 diagnostic (1 error).
2026-08-01T06:52:51Z ERROR basilisk: error processing file path=./broken.py error=syntax error in ./broken.py: Expected an identifier at byte range 20..21

The report body has no entry for broken.py, and the summary says 1
diagnostic. --output json on the same run returns 2 entries. Text and JSON
disagree about what the run found.

The only signal in text mode is a tracing line on stderr, emitted after the
summary and outside the report — so it is lost to 2>/dev/null, to any log
filter, and to anyone reading the report rather than watching the console. With
no other diagnostics present, the text renderer prints nothing at all:

// crates/basilisk-cli/src/main.rs:426
let exit_code = if total == 0 && outcome.failures.is_empty() {
    println!("{}", "All checked. No issues found.".green().bold());
    0
} else if total == 0 {
    0            // <- failures present: prints nothing, reports nothing
}

That branch was clearly added to stop the "All checked. No issues found." lie —
correct instinct, but silence is the other half of the same bug.

2. A user syntax error exits 3 (Internal failure) and masks exit 1
$ basilisk check .          # 1 type error + 1 unparseable file
$ echo $?
3
$ rm broken.py && basilisk check . ; echo $?
1

[CHKARCH-CLI-EXITCODES] defines 3 as Internal failure. A syntax error in
the user's own Python is not internal to Basilisk — it is the most ordinary
finding a checker can have. Two consequences:

  • CI that branches on the documented codes ([CHKARCH-CLI-CI] tells consumers to
    do exactly that) is told the tool broke, not that the code has problems.
    Pipelines that retry or page on 3 will do so on a plain typo.
  • The failure exit overrides the diagnostic exit unconditionally
    (if outcome.failures.is_empty() { diagnostic_exit } else { 3 },
    main.rs:360), so one
    unparseable file hides the fact that real errors were found.

This reproduces on this repository: basilisk check . exits 3 because
basilisk-zed/tests/fixtures/completions.py ends in obj. — a deliberately
incomplete completion fixture. The fixture is correct and must stay invalid; the
handling of it is what is wrong.

3. The position is a byte range, not line:col
syntax error in ./broken.py: Expected an identifier at byte range 20..21

No user can act on byte 20. Every other checker reports file:2:5: invalid syntax. The offset comes straight from ruff_python_parser, which hands us a
TextRange — the line/column conversion the rest of the pipeline already does
for diagnostics is simply never applied here. The entry also carries no code and
therefore no see: URL.

Note the JSON entry anchoring at line 1, column 1 is specified behaviour and
should not change; this is about the message text.

Acceptance

  • Extend [CHKARCH-CLI-OUTPUT-FAILURES] to cover text output, and render
    unanalysable files in the text report with the same standing they have in
    JSON — visible in the body, counted in the summary, on stdout with the
    report rather than as a stderr log line.
  • Decide and specify the exit code for an unanalysable file. 1 matches its
    severity and keeps 3 meaning what the table says. If it must stay
    distinct from 1, it needs its own row in [CHKARCH-CLI-EXITCODES] — not
    a reuse of "Internal failure".
  • Stop a failure from masking a real diagnostic exit.
  • Convert the parser's byte offset to line:col in the message.
  • Regression tests for each: a text-mode assertion mirroring the existing
    cli_binary_tests.rs JSON assertions, a text-vs-JSON agreement test on one
    mixed run, and an exit-code test for failure-only and failure-plus-error
    trees.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in crates/basilisk-cli/src/main.rs around lines 360 and 426, then compare the existing failure handling in crates/basilisk-cli/src/output/json.rs. Read the related cli_binary_tests.rs JSON assertions and the invalid basilisk-zed/tests/fixtures/completions.py fixture. Done means text and JSON agree on failure entries and counts, parser positions are line:col, and failure-only and mixed runs use specified exit codes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
cli, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.