python / python/mypy

Use information from `@overload` to better model narrowing in implementation

Open
#14,666 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature topic-overloads
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Bug Report

There are potentially two issues that I have encountered:

  • Consider a case where we use @overload and, for example, two arguments have narrower types than the signature of the non-@overload-decorated definition. If one of those is a Literal type and we branch on a comparison to the value of that literal type, the other argument isn't narrowed to the type(s) defined in the matching overloaded function(s).
    • This can be worked around by using isinstance(), but it seems redundant.
    • It seems, according to Python's documentation that the current behaviour is sort of incorrect as it states:

      The @overload-decorated definitions are for the benefit of the type checker only, since they will be overwritten by the non-@overload-decorated definition, while the latter is used at runtime but should be ignored by a type checker.

    • But until either the literal comparison occurs or isinstance() is checked on the other argument, it's not possible to narrow, but it should be after.
  • The other issue is that, once we have called isinstance() to narrow the type of a value, a subsequent nested function definition keeps the non-narrowed type when referring to the value.

See the reproducer below which should help clarify the above issues.

To Reproduce

See playground example.

from typing import Literal, overload

@overload
def f(name: Literal["integer"], value: int) -> int: ...

@overload
def f(name: Literal["string"], value: str) -> str: ...

def f(name: Literal["integer", "string"], value: int | str) -> int | str:
    if name == "integer":
        reveal_type(value)
        assert isinstance(value, int)
        reveal_type(value)

        def do_integer() -> int:
            reveal_type(value)  # XXX: Expected `builtins.int`
            return value        # XXX: Expected no error.

        reveal_type(value)

        return do_integer()

    if name == "string":
        reveal_type(value)
        assert isinstance(value, str)
        reveal_type(value)

        def do_string() -> str:
            reveal_type(value)  # XXX: Expected `builtins.str`
            return value        # XXX: Expected no error.

        reveal_type(value)

        return do_string()

Expected Behavior

Something like this mocked up output if the the type of value were narrowed based on the literal of name:

❯ mypy --strict bug.py 
bug.py:11: note: Revealed type is "builtins.int"
bug.py:13: note: Revealed type is "builtins.int"
bug.py:16: note: Revealed type is "builtins.int"
bug.py:19: note: Revealed type is "builtins.int"
bug.py:24: note: Revealed type is "builtins.str"
bug.py:26: note: Revealed type is "builtins.str"
bug.py:29: note: Revealed type is "builtins.str"
bug.py:32: note: Revealed type is "builtins.str"
Success: no issues found in 1 source file

Something like this mocked up output if only the type inside the nested function respected the narrowing performed by the isinstance() calls:

❯ mypy --strict bug.py 
bug.py:11: note: Revealed type is "Union[builtins.int, builtins.str]"
bug.py:13: note: Revealed type is "builtins.int"
bug.py:16: note: Revealed type is "builtins.int"
bug.py:19: note: Revealed type is "builtins.int"
bug.py:24: note: Revealed type is "Union[builtins.int, builtins.str]"
bug.py:26: note: Revealed type is "builtins.str"
bug.py:29: note: Revealed type is "builtins.str"
bug.py:32: note: Revealed type is "builtins.str"
Success: no issues found in 1 source file

Actual Behavior

❯ mypy --strict bug.py 
bug.py:11: note: Revealed type is "Union[builtins.int, builtins.str]"
bug.py:13: note: Revealed type is "builtins.int"
bug.py:16: note: Revealed type is "Union[builtins.int, builtins.str]"
bug.py:17: error: Incompatible return value type (got "Union[int, str]", expected "int")  [return-value]
bug.py:19: note: Revealed type is "builtins.int"
bug.py:24: note: Revealed type is "Union[builtins.int, builtins.str]"
bug.py:26: note: Revealed type is "builtins.str"
bug.py:29: note: Revealed type is "Union[builtins.int, builtins.str]"
bug.py:30: error: Incompatible return value type (got "Union[int, str]", expected "str")  [return-value]
bug.py:32: note: Revealed type is "builtins.str"
Found 2 errors in 1 file (checked 1 source file)

Your Environment

  • Mypy version used: 1.0.0
  • Mypy command-line flags: --strict
  • Mypy configuration options from mypy.ini (and other config files): N/A
  • Python version used: 3.10.9

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked playground example or reproduce the shown bug.py command using mypy --strict, comparing the reveal_type output with the expected output. Investigate overload-based narrowing and narrowing captured by nested functions; done means the requested reveal_type results are produced and both incompatible return errors are gone.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.