dolthub / dolthub/doltlite

VC oracle suites have no pass floor; one suite always exits 0 and doltlite_parity cannot tell a self-comparison

Closed
#2,912 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
268
Forks
18
Avg merge
2h 27m
Merged PRs (30d)
447

Description

Found in a full-repo review at `0ba280f06f`.

The VC oracle helper has no pass floor, and one suite's exit status is unconditionally zero. Both let a suite report success while testing nothing. #2790 fixed this class for the stock-SQL oracles; the same guard was never added to the VC side.

## 1. `vc_oracle_finish` has no floor

`test/lib/stock_oracle_common.sh:42` fails a suite that made no passing, non-empty comparison:

```bash
if [ "$pass" -eq 0 ] || [ "$nonempty" -eq 0 ]; then
fail=$((fail+1))
echo " FAIL: suite needs passing comparisons and non-empty compared output"
```

`test/lib/vc_oracle_common.sh:130` has no equivalent — it only fails when `fail > 0`:

```bash
vc_oracle_finish() {
echo ""
echo "=== Results: $pass passed, $fail failed ==="
if [ "$fail" -gt 0 ]; then
...
exit 1
fi
echo "__SUITE_COMPLETE__"
}
```

So a VC oracle that compares nothing exits 0. The largest VC correctness asset in the repo, the ~500-pair schema-merge matrix, degrades silently to `0 passed, 0 failed, 500 not comparable` if setup starts failing on either engine.

## 2. `vc_oracle_clean_test.sh` passes against an engine that only exits 1

```
$ bash test/vc_oracle_clean_test.sh /usr/bin/false /usr/bin/false
=== Version Control Oracle Tests: dolt_clean ===

=== Results: 11 passed, 0 failed ===
__SUITE_COMPLETE__
rc=0
```

`test/vc_oracle_clean_test.sh:55` joins two possibly-empty strings before asserting:

```bash
vc_oracle_assert_match "$name" "$dl_status|$dl_rows" "$dt_status|$dt_rows"
```

The both-sides-empty guard at `test/lib/vc_oracle_common.sh:56` only rejects a truly empty string, and `"|"` is not empty. Nine cases match `"|" == "|"`; the two `oracle_error()` cases pass because both sides returned a clean non-zero. `dolt_clean` has no effective oracle coverage today.

## 3. `doltlite_stable_catalog_numbers.sh` always exits 0

The last two lines of `test/doltlite_stable_catalog_numbers.sh` are:

```bash
[ "$fail" -eq 0 ]
echo "__SUITE_COMPLETE__"
```

The bare test is not the last statement, so the script's status is the `echo`'s. With one check deliberately broken:

```
$ bash broken_catalog.sh ./doltlite | tail -3
Results: 13 passed, 1 failed out of 14 tests
Failed: hash_shape
__SUITE_COMPLETE__
$ echo $? -> 0
```

The completion guard is satisfied too, so CI sees a pass. This suite pins the canonical-catalog invariant, a repeat offender (#1596, #2285), so it is a bad one to have unenforced. Four other suites put the same bare test on the *last* line and are correct.

## 4. `doltlite_parity.sh` cannot tell a self-comparison

`test/doltlite_parity.sh:3` hardcodes `SQLITE3=./sqlite3` and never calls `assert_stock_reference.sh`:

```
$ cp build/doltlite selfcmp/doltlite && cp build/doltlite selfcmp/sqlite3
$ cd selfcmp && bash test/doltlite_parity.sh ./doltlite
Results: 119 passed, 0 failed out of 119 tests
```

Identical output to a run against a real stock reference. Today's CI reference *is* genuinely stock — I built `make DOLTLITE_PROLLY=0 sqlite3` in a directory that had already built doltlite and it produced a real stock binary (0 prolly symbols, SQLite header, passes `assert_stock_reference.sh`) — so this is latent rather than live. But `test/build_stock_reference.sh`'s own header records that CI has previously shipped a reference sharing doltlite's storage, "which made every comparison against it pass by construction". The guard exists and is used elsewhere (`sql_differential_test.sh:50-61` runs `assert_stock_reference.sh`, `assert_doltlite_engine.sh` *and* a `samefile` check); the one suite actually named "parity" has none of it.

## Fix

Make the floor a property of the harness rather than a habit of each suite:

- give `vc_oracle_finish` the same `pass==0 || nonempty==0` failure as `stock_oracle_finish`, and fail the schema-merge matrix when `skipped` exceeds a pinned ceiling instead of only printing it;
- assert the two components separately in `vc_oracle_clean_test.sh` (or make `vc_oracle_assert_match` reject a value made only of separators);
- swap the last two lines of `doltlite_stable_catalog_numbers.sh`;
- add the three-line reference guard to `doltlite_parity.sh` and the other suites that take a `$SQLITE3` without checking it (`doltlite_attach_sqlite.sh`, `doltlite_attach_write_matrix.sh`, `doltlite_pragma_{auto_vacuum,journal_mode,wal_checkpoint}.sh`, `doltlite_gc_scale.sh`).

And the general guard: a CI job that runs every suite against sabotaged engines (`/usr/bin/true`, `/usr/bin/false`, a silent wrapper, a fixed-string wrapper) and fails if any reports success. A sweep of all 71 `vc_oracle_*` and 24 `oracle_*` suites that way is what found items 1 and 2 above; the stock-oracle class is genuinely fixed and `stock_oracle_harness_test.sh` already re-proves it on every run, which is the pattern to generalize. This class has now been filed five times in three weeks (#2780, #2790, #2791, #2857, #2865) — it needs a mechanism, not another fix.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with test/lib/vc_oracle_common.sh and compare vc_oracle_finish with test/lib/stock_oracle_common.sh:42. Then inspect the named VC and doltlite suites, especially vc_oracle_clean_test.sh, doltlite_stable_catalog_numbers.sh, and doltlite_parity.sh, and run their existing harness tests with sabotaged engines. Done means no-op comparisons and self-comparisons fail, suite exit statuses are reliable, and the relevant CI checks cover the guards.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, sqlite
Domain
ci-cd, databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.