python / python/mypy

Implicit type aliases not recognised with PEP 604 + import cycles

Open
#11,887 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug priority-0-high topic-pep-604 topic-type-alias
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR metrics pending

Description

This bug was originally surfaced by https://github.com/python/typeshed/pull/6612. It is hard to tell if it is the only bug that was revealed by that PR, or if there are other PEP 604-related bugs in mypy.

Top-line summary

If a single line in typeshed/stdlib/_typeshed/__init__.pyi is changed from using Union to PEP 604-style syntax, a large number of errors are output by mypy:

--- a/stdlib/_typeshed/__init__.pyi
+++ b/stdlib/_typeshed/__init__.pyi
@@ -189,7 +189,7 @@ ReadOnlyBuffer = bytes  # stable
 # for it. Instead we have to list the most common stdlib buffer classes in a Union.
 WriteableBuffer = Union[bytearray, memoryview, array.array[Any], mmap.mmap, ctypes._CData]  # stable
 # Same as _WriteableBuffer, but also includes read-only buffer types (like bytes).
-ReadableBuffer = Union[ReadOnlyBuffer, WriteableBuffer]  # stable
+ReadableBuffer = ReadOnlyBuffer | WriteableBuffer  # stable
Mypy errors 
stdlib\io.pyi:44: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:44: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:54: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:54: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:61: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:61: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:74: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:74: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\io.pyi:104: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:104: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:52: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:52: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:53: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:53: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:55: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:55: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\mmap.pyi:64: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\mmap.pyi:64: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ctypes\__init__.pyi:89: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ctypes\__init__.pyi:89: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\builtins.pyi:672: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\builtins.pyi:672: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:9: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:9: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:10: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:10: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:11: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:11: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:23: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:23: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:24: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:24: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\struct.pyi:25: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\struct.pyi:25: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\lzma.pyi:90: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\lzma.pyi:90: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:19: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:19: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:30: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:30: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:31: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:31: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:37: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:37: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hmac.pyi:42: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hmac.pyi:42: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:12: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:12: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:16: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:16: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:19: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:19: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:20: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:20: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:21: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:21: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:22: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:22: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:23: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:23: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:24: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:24: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:25: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:25: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:49: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:49: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:56: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:56: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:60: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:60: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:70: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:70: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:72: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:72: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:89: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:89: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:92: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:92: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:93: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:93: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\hashlib.pyi:94: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\hashlib.pyi:94: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:14: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:14: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:560: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:560: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:561: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:561: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:563: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:563: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\_socket.pyi:565: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\_socket.pyi:565: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\socket.pyi:603: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\socket.pyi:603: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\gzip.pyi:136: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\gzip.pyi:136: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\bz2.pyi:127: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\bz2.pyi:127: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\bz2.pyi:128: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\bz2.pyi:128: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:343: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:343: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:344: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:344: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:346: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:346: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
stdlib\ssl.pyi:348: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\ssl.pyi:348: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases

Investigation

This bug is hard to pin down. I have not managed to reproduce it outside of the typeshed repo.

I have managed to reproduce the bug in a branch of typeshed in which the vast majority of the stdlib has been deleted. This indicates to me that the bug has something to do with an interaction between builtins, _typeshed and mmap. In this "slimmed-down" version of typeshed, I see the following errors from mypy:

stdlib\builtins.pyi:677: error: Name "ReadableBuffer" is not defined
stdlib\io.pyi:5: error: Variable "_typeshed.ReadableBuffer" is not valid as a type
stdlib\io.pyi:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases

Notes on the bug

The bug can be reproduced if ReadableBuffer is defined like this in _typeshed:

import mmap
WriteableBuffer = mmap.mmap
ReadableBuffer = bytes | WriteableBuffer

However...

(1) It cannot be reproduced if ReadableBuffer is defined like this in typeshed:

import mmap
ReadableBuffer = bytes | mmap.mmap

(2) Nor can it be reproduced if ReadableBuffer is defined like this in typeshed:

from mmap import mmap
WriteableBuffer = mmap
ReadableBuffer = bytes | WriteableBuffer

(3) If the definition of ReadableBuffer is moved to builtins, the errors in builtins and io that you see in "slimmed-down typeshed" continue to be reported by mypy.

(4) The errors in the io module that you see in "slimmed-down typeshed" do not exist if io defines its own version of ReadableBuffer, instead of importing the definition from _typeshed or builtins.

Expected behaviour

No error should be reported; the semantics of typing.Union and PEP 604 union-type expressions should be identical.

mypy version

0.930. Reproduced on many Python versions by the typeshed CI; local testing done on Python 3.10.

To reproduce

  1. Either clone the typeshed master branch, and apply the above diff manually, or clone my "slimmed-down typeshed branch".
  2. cd into your clone of the typeshed repo.
  3. Run mypy stdlib/abc.pyi --custom-typeshed-dir . (the precise file you run mypy on is irrelevant).

Cc. @sobolevn, @JukkaL, @Akuli

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure by changing ReadableBuffer in typeshed/stdlib/_typeshed/init.pyi from Union syntax to PEP 604 syntax, then run mypy against the affected stubs and inspect the reported errors. Trace how the alias is resolved through the import cycle; done means PEP 604 aliases are recognised without the invalid-type errors shown, including the affected stdlib modules.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.