python / python/mypy

Adding __init__ breaks type inference from __new__

Open
#12,045 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Hello, i'm trying to fix a bug in django type stubs, and i believe i've stubled upon bug in mypy.

Django has following syntax for defining foreign keys:

class Book(models.Model):
    author = models.ForeignKey(Author, null=True)

This means author should be Author | None.

Right now django-types are using @override for __init__ with differently typed self parameter:

https://github.com/sbdchd/django-types/blob/90fc353b7d05ec600a4807d8b29343a6538692b1/django-stubs/db/models/fields/related.pyi#L293

@overload
def __init__(
        self: ForeignKey[_M],
       ...
       null: Literal[False],
...
@overload
def __init__(
        self: ForeignKey[Optional[_M]],
        ...
       null: Literal[True],
...

And it works for mypy, but doesn't work for pyright. In mypy it works becouse it uses undocumented feature of typing self, as i discovered here:

Proper, and most concise way of doing this, would be overloading __new__ with args/kwargs and bool: Literal[...], and leaving full annotation with all arguments typed in __init__ (which i attempted here https://github.com/sbdchd/django-types/pull/86).

But, when we try to do this: mypy would ignore overrides in __new__ and will always use non-optional case.

To Reproduce

To reproduce this without much clutter, we can use following code:

# related_field_minimal.pyi
from typing import Generic, Literal, Optional, Type, TypeVar, overload

class Model: ...

_M = TypeVar("_M", bound=Optional[Model])

class ForeignKey(Generic[_M]):
    @overload
    def __new__(
        cls,
        to: Type[_M],
        null: Literal[False] = ...,
    ) -> ForeignKey[_M]: ...
    @overload
    def __new__(
        cls,
        to: Type[_M],
        null: Literal[True],
    ) -> ForeignKey[Optional[_M]]: ...
    def __init__(self, to: Type[_M], null: bool) -> None: ...
# models.py
from typing import Optional
from related_field_minimal import ForeignKey, Model


class Test(Model):
    pass


class BookModel(Model):
    required_field = ForeignKey(to=Test, null=False)
    optional_f1 = ForeignKey(to=Test, null=True)
    optional_f2 = ForeignKey[Test](to=Test, null=True)
    optional_f3 = ForeignKey[Optional[Test]](to=Test, null=True)

reveal_type(BookModel.required_field)
reveal_type(BookModel.optional_f1)
reveal_type(BookModel.optional_f2)
reveal_type(BookModel.optional_f3)

When there is no __init__ in ForeignKey, inferred types are correct, both in mypy and pyright.

models.py:15: note: Revealed type is "related_field_minimal.ForeignKey[models.Test]"
models.py:16: note: Revealed type is "related_field_minimal.ForeignKey[Union[models.Test, None]]"
models.py:17: note: Revealed type is "related_field_minimal.ForeignKey[Union[models.Test, None]]"
models.py:18: note: Revealed type is "related_field_minimal.ForeignKey[Union[models.Test, None]]"

As soon as we add __init__ to our type definition in related_field_minimal.pyi, it becomes incorrect in mypy:

models.py:15: note: Revealed type is "related_field_minimal.ForeignKey[models.Test]"
models.py:16: note: Revealed type is "related_field_minimal.ForeignKey[models.Test]"
models.py:17: note: Revealed type is "related_field_minimal.ForeignKey[models.Test]"
models.py:18: note: Revealed type is "related_field_minimal.ForeignKey[Union[models.Test, None]]"

(pyright still handles this case correctly).

Your Environment

  • Mypy version used: 0.931
  • Mypy command-line flags: None
  • Mypy configuration options from mypy.ini (and other config files): None
  • Python version used: 3.10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the mypy 0.931 reproduction using related_field_minimal.pyi and models.py, comparing results with and without init. Check the revealed types against the expected ForeignKey[Test] and ForeignKey[Optional[Test]] outputs; the issue is done when the overloads remain correct after adding the fully typed init.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.