vx-lang / vx-lang/Vx

145 pass-tier tests assert only `// CHECK: module`, which no output can fail

Open
#412 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted testing
Dominant language
Rust
Stars
14
Forks
2
Avg merge
12h 42m
Merged PRs (30d)
61

Description

Vx#407 was about directives that never ran: 281 CHECK lines under prefixes no RUN line named. Those now execute. This is the other half — directives that run and cannot fail.

145 of the 489 test files carrying any CHECK assert only // CHECK: module. Every module the compiler emits contains that word, so the assertion is satisfied by any output at all, including output from a completely different program. These files gate that vxc exited 0. Nothing about what it emitted is checked.

tier                                  module-only / carrying checks
frontend/pass                              56 / 80
backend/pass                               54 / 131
middle_end/pass                            11 / 45
backend/pass/autodiff                       7 / 8
frontend/pass/formal_verification           6 / 6
backend (top level)                         4 / 4
modules                                     3 / 3
frontend/pass/modules_nested                2 / 3
frontend/pass/modules_basic                 1 / 2
tests/paren_test.vx                         1 / 1
                                          ---
                                          145 / 489

tests/frontend/pass/array_index.vx is the shape of it — array literals, indexing and a for loop, and the only claim made about the lowering is that a module came out:

// RUN: vxc %s --action emit-mlir 2>&1 | FileCheck %s
// CHECK: module

What replacing one looks like

33 of these were done in 8cee3adf. Each states its own call chain with the values threaded between the calls captured, so the dataflow is asserted rather than the presence of names. tests/backend/pass/ffi_fs.vx:

// CHECK-DAG: func.func private @vx_get_temp_file(
// CHECK-DAG: func.func private @vx_file_open(
// CHECK-LABEL: func.func @main() -> i32
// CHECK: %[[V0:.*]] = call @vx_get_temp_file(%{{.*}}) : (!llvm.ptr) -> !llvm.ptr
// CHECK: %[[V1:.*]] = call @vx_file_open(%[[V0]], %{{.*}}) : (!llvm.ptr, i32) -> !llvm.ptr
// CHECK: call @vx_file_write(%[[V1]], %{{.*}}, %{{.*}}) : (!llvm.ptr, !llvm.ptr, i64) -> i64
// CHECK: call @vx_file_drop(%[[V1]]) : (!llvm.ptr) -> i32

That says the handle vx_file_open returned is the one written to and the one dropped. Repointing the drop at %[[V0]] is rejected, which is the property // CHECK: module never had. The externs are CHECK-DAG because the module declares them in a different order than the calls use them.

Doing the rest

A generator that derives the block from the program's own call chain handled 27 of the 33 unattended; the remaining 6 needed hand-writing. So roughly 80% of the 145 is mechanical, and the value is concentrated in the part that is not.

The trap, and the reason this cannot be a blind sweep: the generated block has to match the codegen path the tier actually exercises. backend/pass runs with --legacy-codegen injected by the runner; frontend/pass runs the default path. tests/frontend/pass/logical_ops.vx was about to assert arith.andi/arith.ori, which is what the AST path emits — but on the default path && and || are cf.cond_br over a result slot, because they short-circuit. Generating from the wrong path produces a file that asserts a lowering the test never produces, and the failure reads as a compiler bug. It now asserts the short-circuit, which is both correct for the path and a stronger claim than the file ever made.

Related: Vx#407 (the directives that never ran), Vx#338 (--check-prefixes silently ignored).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the 145 module-only files under the listed tests tiers and study commit 8cee3adf, especially tests/backend/pass/ffi_fs.vx, for the expected dataflow assertions. Check each tier's codegen path, including --legacy-codegen for backend/pass and the default path for frontend/pass, before generating or writing FileCheck blocks. Done means the weak // CHECK: module assertions are replaced by checks of the behavior each test exercises.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.