python / python/mypy

Collapsed Optionals in TypeVar cause weird error messages

Open
#8,478 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug false-positive priority-1-normal topic-type-variables
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

At a minimum, I'm reporting a bug in an error message.

I have a function, run_tasks, which takes a list of tasks and a worker function, and executes the worker function on almost all the tasks. For some of them, run_tasks will decide to not execute the task.

run_tasks returns List[Optional[Result]] where Result is a TypeVar.

This works until the worker function's returns Optional[x] where Result is now Optional[x] and run_tasks is asked to return List[Optional[Optional[x]]].

Example code:

from typing import (
    TypeVar,
    List,
    Callable,
    Optional,
)

Task = TypeVar("Task")
Result = TypeVar("Result")


def run_tasks(
    tasks: List[Task], worker_fun: Callable[[Task], Result]
) -> List[Optional[Result]]:
    results: List[Optional[Result]] = []

    toggle: bool = True
    todo: Task
    for todo in tasks:
        toggle = not toggle

        if toggle:
            work_result: Result
            work_result = worker_fun(todo)
            results.append(work_result)
        else:
            results.append(None)

    return results


def test() -> None:
    def worker(d: int) -> Optional[int]:
        if d % 2 == 0:
            return None
        else:
            return d + 1

    output: List[Optional[Optional[int]]] = run_tasks(
        [1, 2, 2, 1], worker,
    )
    from pprint import pprint

    pprint(output)
    assert output == [None, None, None, 2]


test()

I get the error message: error: Argument 2 to "run_tasks" has incompatible type "Callable[[int], Optional[int]]"; expected "Callable[[int], int]"

I understand Python can't really represent Optional[Optional[int]]. The problem is in the error message, I think. This code does work correctly, and validates:

def test2() -> None:
    def worker(d: int) -> int:
        return d + 1

    output: List[Optional[int]] = run_tasks(
        [1, 2, 2, 1], worker,
    )
    from pprint import pprint

    pprint(output)
    assert output == [None, 3, None, 2]


test2()

and so someone unfamiliar with the intricacies and details may be confused by the type signature being correctly updated in 2 places, but not validating.

Instead, I sort of expected it to either (a) work, or (b) have an error specific to the case of nested Optional.

$ mypy --version
mypy 0.761

$ python --version
Python 3.7.6

flags: none.

My apologies, but I did not test with mypy from Git master.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided run_tasks reproducer with mypy and compare the nested-Optional diagnostic with the working test2 case. Done means the incompatible Callable error is corrected or replaced with a diagnostic that accurately explains this nested Optional case, with regression coverage for the example.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.