python / python/mypy

Potential issue with dmypy file save status not being updated properly

Open
#17,603 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-daemon
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Bug Report

To be completely honest I am hesitant to open this issue, as I am unable to reproduce it, but it was really strange so I'm going to anyways.
I ran dmypy suggest on line 108, in example below that's in the function pair_encode_stream on the line for char in text: after previously having a successful typecheck with the following command:

dmypy --status-file="/home/<my username>/.idlerc/mypy/dmypy.json" run --timeout=60 --log-file="/home/<my username>/.idlerc/mypy/log.txt" --export-types "/home/<my username>/Desktop/Python/Tools/byte_pair_encoding.py" -- --warn-unused-ignores --show-absolute-path --disallow-untyped-defs --show-traceback --warn-unreachable --show-error-codes --cache-fine-grained --no-implicit-reexport --warn-redundant-casts --show-error-end --cache-dir="/home/<my username>/.idlerc/mypy" --soft-error-limit=-1 --disallow-untyped-calls --strict --no-error-summary --hide-error-context --no-color-output --show-column-numbers

but had since continued adding lines to the program. I hadn't realized as of yet that line 130 would result in a syntax error.
Apon running dmypy suggest, I got this response from it:

{
    "func_name": "pair_decode",
    "line": 105,
    "path": "/home/<my username>/Desktop/Python/Tools/byte_pair_encoding.py",
    "samples": 0,
    "signature": {
        "arg_types": [
            "bytes",
            "typing.Dict[int, bytes]"
        ],
        "return_type": "None"
    }
}

claiming that the function definition start was on line 105 instead of line 103.
My dmypy extension raised an error about being unable to parse the function definition while attempting to add dmypy's suggestion, which I thought was "aw man looks like I have a bug again" and tried running dmypy suggest on line 104, and
this time it returned something similar to this (I did not record original response at the time):

{
    "func_name": "pair_encode",
    "line": 70,
    "path": "/home/<my username>/Desktop/Python/Tools/byte_pair_encoding.py",
    "samples": 0,
    "signature": {
        "arg_types": [
            "str"
        ],
        "return_type": "Tuple[bytes, typing.Dict[int, bytes]]"
    }
}

notibly being the proper definition start for pair_encode_stream instead of pair_decode

To Reproduce

Exact state of the file when issue was happening

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

"""TITLE - DESCRIPTION"""

# Programmed by CoolCat467

from __future__ import annotations

# TITLE - DESCRIPTION
# Copyright (C) 2024  CoolCat467
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.

__title__ = "Byte pair encoding AKA digram coding"
__author__ = "CoolCat467"
__version__ = "0.0.0"
__license__ = "GNU General Public License Version 3"


from collections import Counter
from collections.abc import Generator, Sequence, Buffer
from typing import TypeVar, TypeGuard
import string

T = TypeVar("T")


def get_groups(data: Sequence[T], n: int = 2) -> Generator[Sequence[T], None, None]:
    if len(data) < n:
        raise ValueError(f"Cannot be length less than {n}")
    for i in range(len(data) - n):
        yield data[i:i+n]
    yield data[-n:]


def get_pairs(string: str) -> Generator[str, None, None]:
    if len(string) < 2:
        raise ValueError("Cannot be length less than 2")
    buffer = [string[0], string[1]]
    for char in string[2:]:
        yield "".join(buffer)
        buffer[0] = buffer[1]
        buffer[1] = char
    yield "".join(buffer)
    

def prove_is_bytes(value: Sequence[int]) -> TypeGuard[bytes]:
    return isinstance(value, bytes)


def prove_bytes_counter(value: Counter[Sequence[int]]) -> TypeGuard[Counter[bytes]]:
    for key in value:
        if not prove_is_bytes(key):
            return False
    return True


def pair_encode(text: str) -> tuple[bytes, dict[int, bytes]]:
    transformed = text.encode("utf-8")
    usable = set(range(256)) - set(transformed)
    encode_map: dict[int, bytes] = {}
    while True:
        pairs = Counter(get_groups(transformed, 2))
        if not prove_bytes_counter(pairs):
            break
        most_common, times = pairs.most_common(1)[0]
        if times < 2:
            break
        replace = usable.pop()
        encode_map[replace] = most_common
        transformed = transformed.replace(most_common, replace.to_bytes())
    return transformed, encode_map


def pair_encode_stream(text: str) -> Generator[tuple[int, bytes] | bytes, None, None]:
    transformed = text.encode("utf-8")
    usable = set(range(256)) - set(transformed)
    while True:
        pairs = Counter(get_groups(transformed, 2))
        if not prove_bytes_counter(pairs):
            break
        most_common, times = pairs.most_common(1)[0]
        if times < 2 or not usable:
            break
        replace = usable.pop()
        yield replace, most_common
        transformed = transformed.replace(most_common, replace.to_bytes())
    yield transformed


def pair_decode(transformed: bytes, decode_map: dict[int, bytes]):
    text = transformed
    matched = True
    while matched:
        matched = False
        for char in text:
            if char not in decode_map:
                continue
            matched = True
            text = text.replace(char.to_bytes(), decode_map[char])
    return text.decode("utf-8")


# types: misc error: The return type of a generator function should be "Generator" or one of its supertypes
def test() -> None:
    yield 3
    yield 4
    return 5

def run() -> None:
    "Run program"
    transformed, encode_map = pair_encode("aaabdaaabac")
    print((transformed, encode_map))
    # types: func-returns-value error: "pair_decode" does not return a value (it only ever returns None)
    print(f'{pair_decode(transformed, encode_map) = }')
# types:    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    for value in pair_encode_stream("waffles are all the rage in mississippi"):
        



# types: syntax error: expected an indented block after 'for' statement on line 129
# types: syntax error: expected an indented block after 'for' statement on line 130
if __name__ == "__main__":
    print(f"{__title__} v{__version__}\nProgrammed by {__author__}.\n")
    run()

And I have no idea if this is relevant, but then apon realizing this is a dmypy bug and copying the program to a different folder and attempting to reproduce this,
the dmypy daemon crashed and apon restarting it and checking the file it told me about the syntax issue.

This is the most recent traceback in log.txt, specified in run arguments.
To be honest not sure if this is related though.

Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 236, in serve
  File "mypy/dmypy_server.py", line 285, in run_command
  File "mypy/dmypy_server.py", line 353, in cmd_run
  File "mypy/dmypy_server.py", line 432, in check
  File "mypy/dmypy_server.py", line 698, in fine_grained_increment_follow_imports
  File "mypy/server/update.py", line 267, in update
  File "mypy/server/update.py", line 369, in update_one
  File "mypy/server/update.py", line 452, in update_module
  File "mypy/server/update.py", line 881, in propagate_changes_using_dependencies
  File "mypy/server/update.py", line 1025, in reprocess_nodes
  File "mypy/checker.py", line 535, in check_second_pass
  File "mypy/checker.py", line 540, in check_partial
  File "mypy/checker.py", line 554, in check_top_level
  File "mypy/nodes.py", line 1351, in accept
  File "mypy/checker.py", line 2934, in visit_assignment_stmt
  File "mypy/checker.py", line 2974, in check_type_alias_rvalue
  File "mypy/checkexpr.py", line 5832, in accept
  File "mypy/errors.py", line 1269, in report_internal_error
  File "mypy/checkexpr.py", line 5830, in accept
  File "mypy/nodes.py", line 2112, in accept
  File "mypy/checkexpr.py", line 3379, in visit_op_expr
  File "mypy/checkexpr.py", line 5832, in accept
  File "mypy/errors.py", line 1269, in report_internal_error
  File "mypy/checkexpr.py", line 5830, in accept
  File "mypy/nodes.py", line 2714, in accept
  File "mypy/checkexpr.py", line 4730, in visit_type_alias_expr
  File "mypy/checkexpr.py", line 4794, in alias_type_in_runtime_context
  File "mypy/checker.py", line 6939, in named_generic_type
  File "mypy/checker.py", line 6946, in lookup_typeinfo
  File "mypy/checker.py", line 7024, in lookup_qualified
KeyError: 'types'

Your Environment

  • Mypy version used: dmypy 1.11.0
  • Mypy command-line flags: See above
  • Mypy configuration options from mypy.ini (and other config files): No other configuration files active
  • Python version used: Python 3.12.0 (main, Apr 28 2024, 22:42:26) [GCC 13.2.0] on linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the dmypy run and suggest behavior with the supplied Python file and command flags, then inspect mypy/dmypy_server.py and the update path in mypy/server/update.py. The traceback also points to checker.py and checkexpr.py. Done means the reported stale status or incorrect suggestion is reproduced and its handling is covered without the daemon crash described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.