python / python/typeshed

Pygments `Formatter.__init__` overloads do not work as intended

Open
#7,436 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.1k
Forks
2.1k
Avg merge
1d 19h
Merged PRs (30d)
82

Description

In #6819 pygments.formatter.Formatter was made generic to improve the type safety:

_T = TypeVar("_T", str, bytes)

class Formatter(Generic[_T]):
    @overload
    def __init__(self: Formatter[str], *, encoding: None = ..., outencoding: None = ..., **options) -> None: ...
    @overload
    def __init__(self: Formatter[bytes], *, encoding: str, outencoding: None = ..., **options) -> None: ...
    @overload
    def __init__(self: Formatter[bytes], *, encoding: None = ..., outencoding: str, **options) -> None: ...

Apparently this was never tested, since it does not actually work:

from pygments.formatters.html import HtmlFormatter

reveal_type(HtmlFormatter())
reveal_type(HtmlFormatter(encoding='utf-8'))
$ mypy foo.py
foo.py:3: note: Revealed type is "pygments.formatters.html.HtmlFormatter[builtins.str*]"
foo.py:4: note: Revealed type is "pygments.formatters.html.HtmlFormatter[builtins.str*]"
$ pyright foo.py
  /tmp/foo.py:3:13 - information: Type of "HtmlFormatter(encoding='utf-8')" is "HtmlFormatter[Unknown]"
  /tmp/foo.py:4:13 - information: Type of "HtmlFormatter()" is "HtmlFormatter[Unknown]"

The intention of the overloads is that HtmlFormatter(encoding='utf-8') should be of type HtmlFormatter[builtins.bytes].

I am not sure if this is a mypy bug or if such overload inference should even work in the first place (given that pyright also infers the type variable to be Unknown: https://github.com/microsoft/pyright/issues/3146).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Formatter overloads introduced in #6819 and the HtmlFormatter entry point. Reproduce the two reveal_type examples with mypy and pyright, then determine whether the overload declarations or checker behavior prevents encoding='utf-8' from inferring HtmlFormatter[bytes]. Done means the intended inferred types are supported or the limitation is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.