Improving support for Annotated[Type, ...] annotations

When writing plugins for code which uses Annotated[Type, an_obj, ..., another_obj] types dynamically, it would be nice to be able to access these annotations in order to either (a) perform type-checking on them, and/or (b) use those dynamic annotations to correctly infer the existence or types of other objects. I'm a total newbie to the mypy internals, but I think this is not currently possible. (See also issue #10872, and PRs #9625, #10777.) It would be fantastic if mypy could support the proper treatment of later parameters of Annotated somehow.

If I understand the current situation correctly, by the time the plugin gets a hold of the type annotation of something which was Annotated, even the unanalyzed_type, mypy will have run the TypeConverter on all of the parameters of the Annotated[...] annotation. This unfortunately erases most of the information in them, rendering them useless to the plugin. Note that PEP 593 states

Annotated is parameterized with a type and an arbitrary list of Python values that represent the annotations.

so it is in principle wrong to treat all of the parameters of Annotated[...] as types anyway.

However, I presume the parser cannot easily tell whether a given Subscript node in the AST is an Annotated[...] field in the relevant part of fastparse.py where this type conversion happens. That is, it cannot really distinguish between dict[str, "Something"] and Annotated[str, "Something"] in order to treat these cases differently (i.e. the first refers to a type Something which happens to be in a string, and the second simply contains a string "Something").

I'm not familiar enough with mypy internals to know what the best solution is. The fundamental tension that mypy wants to do type conversion directly on the AST, but it's not until semantic analysis starts that it knows which nodes are Annotated. I can see that during semantic analysis, expr_to_unanalyzed_type is used. Given that this is possible, it seems to me like the most natural thing is if the AST parsing stage does not do type conversion on Subscript parameters, but instead converts them to Expressions which are later converted to types if appropriate. Presumably this comes at some performance cost, though, in the more common cases of parameters being types (dict[str, int] or list[str] etc.).

Any thoughts would be fantastic.

Start by reading the Annotated handling in mypy/fastparse.py around lines 1863-1898 and the semantic-analysis path through expr_to_unanalyzed_type, then review issues #10872 and PRs #9625 and #10777. Define tests showing that arbitrary Annotated metadata remains available to plugins while ordinary generic parameters continue to be treated as types.