python / python/mypy

Overlapping overloads in second-order functions cause returned value to be Any

Open
#13,436 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-overloads
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Bug Report

Changing a parameter of a function from a determined type (e.g. int, str, etc.) to Any makes the return type of a second-order function applied on the initial function Any in some cases.

I originally found this issue using the pytest library (see in the wild section below).

To Reproduce
from typing import Any, Callable, overload, TypeVar

C = TypeVar("C", bound=Callable[..., object])

#
## the second-order function
#

@overload
def wrapper(arg: C) -> C:
    ...

@overload
def wrapper(arg: object) -> object:
    ...

def wrapper(arg: object) -> object:
    return arg  # we don't care

#
## the "working" case
#

def case_a(x: int) -> None:
    pass

reveal_type(case_a)           # Revealed type is "def (x: builtins.int)"
reveal_type(wrapper(case_a))  # Revealed type is "def (x: builtins.int)"

#
## the "issue" case: only the type of the `x` parameter has changed
#

def case_b(x: Any) -> None:
    pass

reveal_type(case_b)           # Revealed type is "def (x: Any)
reveal_type(wrapper(case_b))  # Revealed type is "Any"
Expected Behavior

reveal_type should give the same result for case_b and wrapper(case_b) (just like it did for case_a and wrapper(case_a)).

Actual Behavior

wrapper(case_b) is thought to have type Any

In the wild (pytest)
from typing import Any, Callable, TYPE_CHECKING, Union

import pytest

@pytest.mark.parametrize("obj", (42, "42"))
def test_bug0(obj: Union[int, str]) -> None:
    assert len(str(obj)) < 20

@pytest.mark.parametrize("obj", (42, "42"))
def test_bug1(obj: Any) -> None:
    assert len(str(obj)) < 20

@pytest.mark.parametrize("obj", (int, float))
def test_bug2(obj: Callable[[str], float]) -> None:
    assert len(obj.__name__) < 20

@pytest.mark.parametrize("obj", (int, float))
def test_bug3(obj: Callable[..., Any]) -> None:
    assert len(obj.__name__) < 20

if TYPE_CHECKING:
    reveal_type(test_bug0)  # Revealed type is "def (obj: Union[builtins.int, builtins.str])"
    reveal_type(test_bug1)  # Revealed type is "Any"
    reveal_type(test_bug2)  # Revealed type is "def (fct: def (builtins.str) -> builtins.float)"
    reveal_type(test_bug3)  # Revealed type is "Any"

We can see that test_bug0 and test_bug2 are well-typed, but test_bug1 and test_bug3 are typed as Any, which cause an error each (with --disallow-any-decorated):

$ mypy --disallow-any-decorated bug.py
bug.py:10: error: Function is untyped after decorator transformation
bug.py:18: error: Function is untyped after decorator transformation
bug.py:22: note: Revealed type is "def (obj: Union[builtins.int, builtins.str])"
bug.py:23: note: Revealed type is "Any"
bug.py:24: note: Revealed type is "def (fct: def (builtins.str) -> builtins.float)"
bug.py:25: note: Revealed type is "Any"
Found 2 errors in 1 file (checked 1 source file)

Here is the relevant parts of the source code of pytest for easy reference:

Markable = TypeVar("Markable", bound=Union[Callable[..., object], type])

class MarkDecorator:
    # Type ignored because the overloads overlap with an incompatible
    # return type. Not much we can do about that. Thankfully mypy picks
    # the first match so it works out even if we break the rules.
    @overload
    def __call__(self, arg: Markable) -> Markable:  # type: ignore[misc]
        pass

    @overload
    def __call__(self, *args: object, **kwargs: object) -> "MarkDecorator":
        pass

    def __call__(self, *args: object, **kwargs: object):
        pass  # redacted

# pytest.mark.parametrize returns a MarkDecorator instance
Your Environment
  • Mypy version used: 0.971
  • Mypy command-line flags: none, except for the pytest case (see above)
  • Mypy configuration options from mypy.ini (and other config files): none
  • Python version used: 3.10
  • Operating system and version: Arch Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided wrapper/case_a/case_b reproducer and the pytest MarkDecorator.call example from src/_pytest/mark/structures.py. Trace overload resolution for the Any-parameter callable and compare the reveal_type results. Done means wrapper(case_b), test_bug1, and test_bug3 retain their callable signatures instead of becoming Any, while the existing cases remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.