Conformance testing for linter?

We've noticed while making changes to AstKind (https://github.com/oxc-project/oxc/issues/11490) that it's often possible to introduce bugs in the linter without any tests failing.

The tests for rules are often taken directly from the ESLint/Unicorn/etc rule which they aim to replicate. They often:

1. Are quite a minimal set of test cases.
2. Only cover edge cases which are potentially problematic in the rule's JS implementation. But, given the large differences between Rust and JS, and ESLint and Oxc, these don't necessarily correspond to likely pitfalls in our Rust implementations of these same rules.

I assume that for the vast majority of rules, the Oxc implementation of the rule aims to replicate the behavior of the original ESLint rule exactly (there are some exceptions, where we diverge from ESLint slightly, where we think we can do "better").

Could we add some kind of conformance testing for linter, along the lines of what we have for parser, transformer etc?

• Take a set of test fixtures (either e.g. Test262, or a selection of real-world code taken from Github).
• Run ESLint on these test fixtures, with all rules that Oxlint provides enabled.
• Conformance tester run Oxlint on these same fixtures, with all rules enabled.
• Conformance check that Oxlint produces same errors as ESLint.

The ESLint results could be generated and stored in a separate repo, which is included in Oxc as a submodule (similar to https://github.com/oxc-project/acorn-test262). That way we don't have to run ESLint every time we run conformance.

I imagine comparing the outputs might be a little tricky. Oxlint likely produces different (better?) error messages in some cases, and may use a different span for some errors from ESLint. If we compare Oxlint's and ESLint's output directly, there'd likely be a lot of erroneous mismatches. But we could probably at least compare the number of errors flagged by Oxlint and ESLint for each rule. Those error counts should match if Oxlint's implementation of the rule replicates ESLint's behavior.

Start by reviewing the existing conformance testing for parser and transformer, along with the acorn-test262 setup mentioned in the issue. Compare how ESLint and Oxlint could run against shared fixtures and how results might be stored and compared. Done means a defined fixture source, reproducible runs, and a comparison that accounts for differing messages or spans.