akiomik / akiomik/mado

mado and mdl disagree on 36 of the 52 acceptance fixtures

未關閉
#401 5 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
bug
主要語言
Rust
星號
407
分支
12
平均合併
5 小時 45 分鐘
30 天內合併 PR
30

描述

The acceptance comparison is not clean, and nothing recorded that. Raised while
fixing #396, which repaired the setup script that prepares this corpus.

## What the comparison shows

`scripts/acceptance/test.sh` runs `mdl` and `mado check --output-format=mdl`
over the same fixtures, each under one fixed config (`scripts/acceptance/.mdlrc`
and the repository's `mado.toml`), and writes `tmp/mdl.txt` and `tmp/mado.txt`.
Diffing those two files on a corpus prepared by `setup.sh`:

- **36 of 52 fixtures differ.**
- 103 findings mado reports and mdl does not.
- 30 findings mdl reports and mado does not.

Both files use the same `path:line: MDxxx text` form and absolute paths, so
these are real disagreements, not formatting.

| mado only | | mdl only | |
| --- | --- | --- | --- |
| MD007 | 40 | MD031 | 20 |
| MD041 | 31 | MD007 | 3 |
| MD003 | 6 | MD032 | 2 |
| MD025 | 5 | MD022 | 2 |
| MD031 | 4 | MD028 | 1 |
| MD022 | 3 | MD024 | 1 |
| MD040, MD038, MD020, MD006, MD004, MD002 | 2 each | MD013 | 1 |
| MD028, MD014 | 1 each | | |

`MD007`, `MD041` and `MD031` are all listed in `.mdlrc`, so this is not a
question of one tool having a rule the other was not asked for. MD031 is the
interesting one: it appears on **both** sides, so the two tools are finding it
in different places rather than one of them missing it.

## Why it is worth a look now

The corpus these numbers come from is what `setup.sh` is supposed to produce:
the fixtures upstream pairs with an `X_style.rb` are set aside, because those
are written against a rule set of their own that `test.sh` cannot apply. So
this is the intended comparison, and it is the one that disagrees.

Two things to separate before reading much into the totals:

- Whether `.mdlrc` and `mado.toml` really do ask for the same thing. They are
written in different formats and were not derived from one another.
- Which differences are mado bugs, which are mdl bugs, and which are documented
intentional divergences. Nothing in the repository currently says.

## Note on how this went unnoticed

`setup.sh`'s exclusion loop has never run in the committed script — see #396 —
so anyone cloning fresh compares all 88 fixtures instead of these 52, on a
corpus that includes documents designed for other rule sets. The numbers above
come from a checkout where the exclusion had been applied by hand before the
script was committed.

`test.sh` also has no `diff` and no assertion: it writes two files and stops.
So there has never been anything to notice, which is #400's subject from a
different angle.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。