VC oracle suites have no pass floor; one suite always exits 0 and doltlite_parity cannot tell a self-comparison
- Dominant language
- C
- Stars
- 268
- Forks
- 18
- Avg merge
- 2h 27m
- Merged PRs (30d)
- 447
Description
Found in a full-repo review at `0ba280f06f`.
The VC oracle helper has no pass floor, and one suite's exit status is unconditionally zero. Both let a suite report success while testing nothing. #2790 fixed this class for the stock-SQL oracles; the same guard was never added to the VC side.
## 1. `vc_oracle_finish` has no floor
`test/lib/stock_oracle_common.sh:42` fails a suite that made no passing, non-empty comparison:
```bash
if [ "$pass" -eq 0 ] || [ "$nonempty" -eq 0 ]; then
fail=$((fail+1))
echo " FAIL: suite needs passing comparisons and non-empty compared output"
```
`test/lib/vc_oracle_common.sh:130` has no equivalent — it only fails when `fail > 0`:
```bash
vc_oracle_finish() {
echo ""
echo "=== Results: $pass passed, $fail failed ==="
if [ "$fail" -gt 0 ]; then
...
exit 1
fi
echo "__SUITE_COMPLETE__"
}
```
So a VC oracle that compares nothing exits 0. The largest VC correctness asset in the repo, the ~500-pair schema-merge matrix, degrades silently to `0 passed, 0 failed, 500 not comparable` if setup starts failing on either engine.
## 2. `vc_oracle_clean_test.sh` passes against an engine that only exits 1
```
$ bash test/vc_oracle_clean_test.sh /usr/bin/false /usr/bin/false
=== Version Control Oracle Tests: dolt_clean ===
=== Results: 11 passed, 0 failed ===
__SUITE_COMPLETE__
rc=0
```
`test/vc_oracle_clean_test.sh:55` joins two possibly-empty strings before asserting:
```bash
vc_oracle_assert_match "$name" "$dl_status|$dl_rows" "$dt_status|$dt_rows"
```
The both-sides-empty guard at `test/lib/vc_oracle_common.sh:56` only rejects a truly empty string, and `"|"` is not empty. Nine cases match `"|" == "|"`; the two `oracle_error()` cases pass because both sides returned a clean non-zero. `dolt_clean` has no effective oracle coverage today.
## 3. `doltlite_stable_catalog_numbers.sh` always exits 0
The last two lines of `test/doltlite_stable_catalog_numbers.sh` are:
```bash
[ "$fail" -eq 0 ]
echo "__SUITE_COMPLETE__"
```
The bare test is not the last statement, so the script's status is the `echo`'s. With one check deliberately broken:
```
$ bash broken_catalog.sh ./doltlite | tail -3
Results: 13 passed, 1 failed out of 14 tests
Failed: hash_shape
__SUITE_COMPLETE__
$ echo $? -> 0
```
The completion guard is satisfied too, so CI sees a pass. This suite pins the canonical-catalog invariant, a repeat offender (#1596, #2285), so it is a bad one to have unenforced. Four other suites put the same bare test on the *last* line and are correct.
## 4. `doltlite_parity.sh` cannot tell a self-comparison
`test/doltlite_parity.sh:3` hardcodes `SQLITE3=./sqlite3` and never calls `assert_stock_reference.sh`:
```
$ cp build/doltlite selfcmp/doltlite && cp build/doltlite selfcmp/sqlite3
$ cd selfcmp && bash test/doltlite_parity.sh ./doltlite
Results: 119 passed, 0 failed out of 119 tests
```
Identical output to a run against a real stock reference. Today's CI reference *is* genuinely stock — I built `make DOLTLITE_PROLLY=0 sqlite3` in a directory that had already built doltlite and it produced a real stock binary (0 prolly symbols, SQLite header, passes `assert_stock_reference.sh`) — so this is latent rather than live. But `test/build_stock_reference.sh`'s own header records that CI has previously shipped a reference sharing doltlite's storage, "which made every comparison against it pass by construction". The guard exists and is used elsewhere (`sql_differential_test.sh:50-61` runs `assert_stock_reference.sh`, `assert_doltlite_engine.sh` *and* a `samefile` check); the one suite actually named "parity" has none of it.
## Fix
Make the floor a property of the harness rather than a habit of each suite:
- give `vc_oracle_finish` the same `pass==0 || nonempty==0` failure as `stock_oracle_finish`, and fail the schema-merge matrix when `skipped` exceeds a pinned ceiling instead of only printing it;
- assert the two components separately in `vc_oracle_clean_test.sh` (or make `vc_oracle_assert_match` reject a value made only of separators);
- swap the last two lines of `doltlite_stable_catalog_numbers.sh`;
- add the three-line reference guard to `doltlite_parity.sh` and the other suites that take a `$SQLITE3` without checking it (`doltlite_attach_sqlite.sh`, `doltlite_attach_write_matrix.sh`, `doltlite_pragma_{auto_vacuum,journal_mode,wal_checkpoint}.sh`, `doltlite_gc_scale.sh`).
And the general guard: a CI job that runs every suite against sabotaged engines (`/usr/bin/true`, `/usr/bin/false`, a silent wrapper, a fixed-string wrapper) and fails if any reports success. A sweep of all 71 `vc_oracle_*` and 24 `oracle_*` suites that way is what found items 1 and 2 above; the stock-oracle class is genuinely fixed and `stock_oracle_harness_test.sh` already re-proves it on every run, which is the pattern to generalize. This class has now been filed five times in three weeks (#2780, #2790, #2791, #2857, #2865) — it needs a mechanism, not another fix.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with test/lib/vc_oracle_common.sh and compare vc_oracle_finish with test/lib/stock_oracle_common.sh:42. Then inspect the named VC and doltlite suites, especially vc_oracle_clean_test.sh, doltlite_stable_catalog_numbers.sh, and doltlite_parity.sh, and run their existing harness tests with sabotaged engines. Done means no-op comparisons and self-comparisons fail, suite exit statuses are reliable, and the relevant CI checks cover the guards.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, sqlite
- Domain
- ci-cd, databases, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100