python / python/typeshed

Definition of `str.split`, `str.rsplit` and `str.splitlines` results in false positive errors

Open
#10,887 9 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stubs: false positive
Dominant language
Python
Stars
5.1k
Forks
2.1k
Avg merge
1d 19h
Merged PRs (30d)
82

Description

I recently fixed a bug in pyright that caused a false negative when attempting to assign a list[LiteralString] to a list[str]. As we know, the type parameter for list is invariant, so these types are not consistent.

from typing import LiteralString
v1: list[LiteralString] = ["a", "b"]
v2: list[str] = v1  # Should be a type violation

With this bug fix now in place, the definitions for the methods str.split, str.rsplit and str.splitlines are problematic. These three methods are defined using overloaded signatures, and the first overload returns list[LiteralString]. This means the following code now results in a type violation.

def process(x: list[str]) -> str:
    return " ".join(x)

process("1,2,3".split(","))  # Type violation

This is going to generate a lot of new type errors in existing code bases.

I initially attempted to work around this by adding a heuristic in pyright's overload matching logic so it deprioritized the first overload in this case, but this heuristic broke other code and really isn't sound. I've therefore decided to remove the heuristic.

I don't have any good ideas about how to fix this, but I wanted to open the issue to start the discussion to see if anyone had a suggestion. If we're going to fix it, I think the fix needs to be in the typeshed stubs. I don't see a way to fix this in a type checker — at least while remaining true to the type system.

One possible solution is to omit the LiteralString overloads for these three methods. This would eliminate the type violation error in the common case, but it would introduce new errors in cases where callers expect to see a return result of list[LiteralString].

Note that this doesn't currently affect mypy users because mypy has not implemented support for LiteralString.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the overloaded typeshed definitions for str.split, str.rsplit, and str.splitlines, focusing on their LiteralString return types. Use the examples in the issue to reproduce the list invariance errors and assess the alternatives discussed. Done means resolving the false positives without undermining valid LiteralString results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.