julia-script / julia-script/silk
tooling: make every capability-gated test visible when it is skipped
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 48
- Forks
- 0
- Avg merge
- 4h 49m
- Merged PRs (30d)
- 213
Description
Problem
A local test run can skip an LLVM-dependent assertion without the default reporter showing either
the skip or its reason.
llvmToolchain(...) currently prints a message with console.log and exposes
unavailable(): boolean. Callers return early when it is true. The default reporter does not
reliably print that log, and an early return still counts as a passing test. The run can therefore be
green while the guarded assertion never ran.
The WebAssembly capability helper already has the desired shape: it puts the reason in the skipped
test's name and makes CI fail when a capability promised by the pinned environment is missing. The
LLVM helper has not adopted that behavior, so the repository still has two incompatible
capability-gate patterns.
Current behavior
A local host without opt can report only a passing test:
✓ verifies emitted LLVM modules
Tests 1 passed (1)
Nothing in the default output says that the external opt verification did not run.
Expected behavior
The same run must expose one real skipped test and the reason:
↓ verifies emitted LLVM modules — skipped: opt was not found
Tests 1 skipped (1)
In CI, the missing pinned tool is not an environment-dependent skip. It is a failure explaining
which tool is absent and which assertion could not run.
Required behavior
- Every capability or external-tool gate uses one shared test-requirement abstraction.
- A missing local capability produces a skipped test, not an early successful return.
- The default reporter shows what is missing and what consequently did not run.
- The summary's skipped count is non-zero.
- CI fails when a capability or tool promised by the pinned CI environment is absent.
- No guarded assertion is removed or weakened.
Relevant cases
LLVM tool unavailable locally
const requirement = requireCapability(llvmToolchain(['opt']), 'verifies emitted LLVM modules')
it.skipIf(requirement.skip)(requirement.name, () => verifyWithOpt())
The exact API may differ, but the result must be an observable skip.
Pinned CI tool unavailable
The test must fail before reporting success. The failure must identify opt and the verification
that was lost.
Capability available
The original test name and assertions run normally, with no skipped count.
Evidence
test/support/llvmToolchain.tsstill logs locally and returnstruefromunavailable().packages/wasm/test/support/hostCapability.tsalready carries the skip reason in the test name
and converts a missing CI capability into a failure.
Acceptance criteria
- Running an LLVM-dependent suite without the requested tool shows a reporter-visible skipped
test and reason. - The default summary reports at least one skipped test.
- Running the same case with
CI=1fails and names the missing tool and guarded assertion. - LLVM and WebAssembly capability gates use the same shared mechanism.
- Capability-present runs execute every original assertion unchanged.
- Focused tests cover local skip, CI failure, and capability-present execution.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing test/support/llvmToolchain.ts with packages/wasm/test/support/hostCapability.ts, focusing on how skip reasons and missing CI capabilities are represented. Trace the LLVM-dependent tests and add focused coverage for local skips, CI failure, and capability-present execution; done means the reporter shows the skipped test and reason, CI names the missing tool and assertion, and existing assertions still run when available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100