Add a --warn-unused-strictness-exceptions flag

In an ideal world all code passes with mypy’s most strict flag settings. Sadly most strictness flags have many violations in the current codebase. We want all new files to be checked with the most strict flag settings, while allowing existing files to still have violations. This can be done with a blacklist. But we want the set of files with violations (i.e. the blacklist) to gradually decrease. Therefore we want a “ratchet” in place where once a file stops having violations it is removed from the blacklist (so from then on it will remain clean). And we want a separate blacklist per strictness flag.

At Dropbox, for strict_optional we’ve got a ratchet in place, but it’s expensive — it uses a separate mypy build and some scripts. Here’s how it works:

• In our mypy.ini the main (default) section has strict_optional = True
• The ground truth for the blacklist is file-specific exceptions in mypy.ini
• A separate CI build modifies the mypy.ini to erase the file-specific exceptions, then collects errors from mypy, and emits an error only for files that have an exception but no mypy errors
• Users must make this CI build pass before they can land their change, which they do by removing the exceptions from mypy.ini for files that are now clear
• Users are discouraged to add new exceptions to mypy.ini

We'd like to create a similar ratchet for disallow_any = generics, because it masks type checking for cases where people should have written Future[X] (for some type X) but accidentally write Future, which is interpreted as Future[Any]. But this flag has many other violations in existing code (too many to just fix before we turn on the flag), so we need a blacklist, and we want it to be a ratchet.

The simplest solution is to just have another CI build that does the same as what we do for strict_optional but for disallow_any. However if we consider a future with many different strictness flags, it would be nice to have a solution that doesn’t require another CI build per flag.

For flags like disallow_any = generics there is actually a better solution possible. We can make a small change to mypy that tracks two sets of files while it is analyzing the code: the set of files for which the flag is disabled (i.e. the blacklist), and the set of files for which an error would have been issued if the flag were enabled (but wasn’t, because the flag was disabled). At the end of the analysis we subtract the latter set from the former, and the remainder is the list of files that don’t need the flag to be disabled.

There is already a similar feature in mypy, --warn-unused-configs, which does a similar thing for unused config sections (a more serious offense, where a section in mypy.ini references a file that doesn’t exist, or at least isn’t analyzed).

The proposed new feature could take the form of a single new flag, e.g. --warn-unused-strictness-exceptions. It would, in the way sketched above, track exceptions for each strictness flag that is (a) off by mypy default, (b) selectable on a per-file basis, and (c) easily trackable. By the latter condition I mean that there is a small number of places in mypy where an error is generated only if the given strictness flag is enabled. Tracking would only be done for flags that are enabled in the main (default) section of mypy.ini.

I believe that at least the various disallow_any flags added by @ilinum are all easily trackable; some others (e.g. disallow_untyped_calls) are also in this category. Some other strictness flags are not easily trackable, e.g. strict_optional or disallow_unchecked_defs can cause inferred types to change, and then it’s not easy to correlate errors emitted with the value of the flag.