akiomik / akiomik/mado

The development scripts report results without having run anything

未关闭
#400 6 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
ci
主要语言
Rust
星标
409
派生
12
平均合并
5 小时 45 分钟
30 天内合并 PR
30

描述

Split out of #396, which was narrowed to the bugs that stop the scripts from
doing what they say. This collects the other half: every way
`scripts/acceptance` and `scripts/benchmarks` produce an answer when the thing
being measured never ran. None of them fails loudly; each looks like a result.

## The cases

| where | what is missing | what you get |
| --- | --- | --- |
| `acceptance/test.sh` | `$PROJECT_ROOT/tmp`, which is gitignored and which nothing creates | both redirects fail, the second only after paying for a `cargo run` |
| `acceptance/test.sh` | `mdl` (a Ruby gem, vendored nowhere) | empty `mdl.txt`; two empty files compare equal, so mado and mdl "agree" |
| `acceptance/test.sh` | `$DOC_PATH`, until `setup.sh` has run | both files empty, same false agreement |
| `acceptance/test.sh` | a build that fails, or no `cargo` | `cargo run` writes only to stderr, `mado.txt` empty, exit 0 |
| `benchmarks/comparison.sh` | `node_modules`, which is gitignored and which nothing installs | `hyperfine --ignore-failure` publishes `markdownlint` and `markdownlint-cli2` as roughly a thousand times faster than mado |
| `benchmarks/comparison.sh` | mado's binary, when `CARGO_TARGET_DIR` or `build.target-dir` sends it elsewhere | `cargo build` succeeds, hyperfine times a command that cannot start, mado "wins" |
| both | the repository, when run from another directory | `cargo` cannot find the manifest |

The `node_modules` one is the common case, not a corner: `scripts/benchmarks/`
tracks `package.json` and `package-lock.json`, `node_modules` is gitignored,
nothing runs `npm ci`, and README.md documents the benchmark as `setup.sh` then
`comparison.sh`. A fresh clone following the README publishes wrong numbers.

## What a fix has to decide

Not just "add a check". `hyperfine --ignore-failure` is there deliberately,
because the linters legitimately exit non-zero when they find violations —
`mado check` on a file with violations returns 1 — so the scripts cannot simply
gate on exit status. The distinction that has to be drawn is *present and
built* versus *what it reported*, and it has to be drawn for every external
tool the scripts invoke.

Whether `setup.sh` should also run `npm ci`, or the README should gain the
step, is part of the same decision.

## Why the linter would not have caught it

None of these is visible to `shellcheck`. They are facts about
`hyperfine --ignore-failure`'s semantics, about cargo's output path, about
which files `git` restores, and about what an empty file means when two are
compared — not about shell syntax. #398 is worth doing on its own terms, but it
would not have found a single row of the table above.

That is also the argument in #397, applied to these scripts rather than to
`action/entrypoint.sh`: the failure mode here is not that the shell is written
badly, it is that this much conditional external-tool orchestration has no
tests and no type system, and every review round finds another unguarded
dependency. Related: #399, where the same class turned destructive.

贡献指南

打开贡献指南

调研方向

Start with scripts/acceptance/test.sh and scripts/benchmarks/comparison.sh, then read setup.sh and the README.md benchmark instructions. Trace each external dependency and cargo output path, preserving legitimate linter failures while distinguishing missing or unrun tools from reported results. Done means the scripts cannot publish results when prerequisites or measured commands did not run, and the setup instructions match the required dependencies.

由索引模型根据 Issue 内容生成。

评估

技术栈
node.js, ruby, rust, shell
领域
testing-qa, tooling
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
活跃
描述清晰度
基本清楚
新手友好度
42/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。