Conformance tests

I posted this on Slack, and @s0undt3ch suggested that it would be better to file an issue so it won't get lost.

The idea is to focus on adding low to medium effort / high impact tests, that would help to improve feature quality/parity. There are many things in Salt that could be invoked in multiple ways (or are pluggable) and are expected to work identically. Just a few examples:

1. Feature parity between different commands like salt, salt-call, salt-ssh, salt-run salt.*, salt-run ssh.cmd. For example, missing wrappers for salt-ssh: https://github.com/saltstack/salt/issues/50196. What if some automated tool was able to run each module using all these salt-* commands and reported any inconsistencies/crashes?
2. Ensuring that all cli utilies handle identical arguments in the same way, to prevent things like this: https://github.com/saltstack/salt/pull/50974, https://github.com/saltstack/salt/pull/53679
3. Batch and non-batch APIs: https://github.com/saltstack/salt/issues/52762
4. Ensuring that all modules conform to the same interface: https://github.com/saltstack/salt/issues/51809
5. A test to detect accidentally introduced new hard dependencies (https://github.com/saltstack/salt/issues/53570#issuecomment-508691287)
6. A pylint plugin to warn on direct imports: https://github.com/saltstack/salt/pull/50633#issuecomment-443579448
7. Another pylint plugin to detect states that do not respect test=True (this is one of the basic expectations about Salt!). Just a rough example: comm -23 <(find salt/states -name '*.py' | sort -u) <(find salt/states -name '*.py' -exec grep -l -E '__opts__.*test' \{\} \; | sort -u )

These meta-tests (or conformance tests) could automatically check all new code to conform to existing coding conventions/invariants/architectural decisions. Two examples: https://github.com/saltstack/salt/pull/52368 https://github.com/saltstack/salt/pull/51900

I'm sure there is a lot of institutional knowledge that could be transformed into these kinds of tests that will assist in PR reviews.