Potential issue with dmypy file save status not being updated properly
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20.6k
- Forks
- 3.3k
- PR merge metrics
- PR metrics pending
Description
Bug Report
To be completely honest I am hesitant to open this issue, as I am unable to reproduce it, but it was really strange so I'm going to anyways.
I ran dmypy suggest on line 108, in example below that's in the function pair_encode_stream on the line for char in text: after previously having a successful typecheck with the following command:
dmypy --status-file="/home/<my username>/.idlerc/mypy/dmypy.json" run --timeout=60 --log-file="/home/<my username>/.idlerc/mypy/log.txt" --export-types "/home/<my username>/Desktop/Python/Tools/byte_pair_encoding.py" -- --warn-unused-ignores --show-absolute-path --disallow-untyped-defs --show-traceback --warn-unreachable --show-error-codes --cache-fine-grained --no-implicit-reexport --warn-redundant-casts --show-error-end --cache-dir="/home/<my username>/.idlerc/mypy" --soft-error-limit=-1 --disallow-untyped-calls --strict --no-error-summary --hide-error-context --no-color-output --show-column-numbers
but had since continued adding lines to the program. I hadn't realized as of yet that line 130 would result in a syntax error.
Apon running dmypy suggest, I got this response from it:
{
"func_name": "pair_decode",
"line": 105,
"path": "/home/<my username>/Desktop/Python/Tools/byte_pair_encoding.py",
"samples": 0,
"signature": {
"arg_types": [
"bytes",
"typing.Dict[int, bytes]"
],
"return_type": "None"
}
}
claiming that the function definition start was on line 105 instead of line 103.
My dmypy extension raised an error about being unable to parse the function definition while attempting to add dmypy's suggestion, which I thought was "aw man looks like I have a bug again" and tried running dmypy suggest on line 104, and
this time it returned something similar to this (I did not record original response at the time):
{
"func_name": "pair_encode",
"line": 70,
"path": "/home/<my username>/Desktop/Python/Tools/byte_pair_encoding.py",
"samples": 0,
"signature": {
"arg_types": [
"str"
],
"return_type": "Tuple[bytes, typing.Dict[int, bytes]]"
}
}
notibly being the proper definition start for pair_encode_stream instead of pair_decode
To Reproduce
Exact state of the file when issue was happening
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""TITLE - DESCRIPTION"""
# Programmed by CoolCat467
from __future__ import annotations
# TITLE - DESCRIPTION
# Copyright (C) 2024 CoolCat467
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
__title__ = "Byte pair encoding AKA digram coding"
__author__ = "CoolCat467"
__version__ = "0.0.0"
__license__ = "GNU General Public License Version 3"
from collections import Counter
from collections.abc import Generator, Sequence, Buffer
from typing import TypeVar, TypeGuard
import string
T = TypeVar("T")
def get_groups(data: Sequence[T], n: int = 2) -> Generator[Sequence[T], None, None]:
if len(data) < n:
raise ValueError(f"Cannot be length less than {n}")
for i in range(len(data) - n):
yield data[i:i+n]
yield data[-n:]
def get_pairs(string: str) -> Generator[str, None, None]:
if len(string) < 2:
raise ValueError("Cannot be length less than 2")
buffer = [string[0], string[1]]
for char in string[2:]:
yield "".join(buffer)
buffer[0] = buffer[1]
buffer[1] = char
yield "".join(buffer)
def prove_is_bytes(value: Sequence[int]) -> TypeGuard[bytes]:
return isinstance(value, bytes)
def prove_bytes_counter(value: Counter[Sequence[int]]) -> TypeGuard[Counter[bytes]]:
for key in value:
if not prove_is_bytes(key):
return False
return True
def pair_encode(text: str) -> tuple[bytes, dict[int, bytes]]:
transformed = text.encode("utf-8")
usable = set(range(256)) - set(transformed)
encode_map: dict[int, bytes] = {}
while True:
pairs = Counter(get_groups(transformed, 2))
if not prove_bytes_counter(pairs):
break
most_common, times = pairs.most_common(1)[0]
if times < 2:
break
replace = usable.pop()
encode_map[replace] = most_common
transformed = transformed.replace(most_common, replace.to_bytes())
return transformed, encode_map
def pair_encode_stream(text: str) -> Generator[tuple[int, bytes] | bytes, None, None]:
transformed = text.encode("utf-8")
usable = set(range(256)) - set(transformed)
while True:
pairs = Counter(get_groups(transformed, 2))
if not prove_bytes_counter(pairs):
break
most_common, times = pairs.most_common(1)[0]
if times < 2 or not usable:
break
replace = usable.pop()
yield replace, most_common
transformed = transformed.replace(most_common, replace.to_bytes())
yield transformed
def pair_decode(transformed: bytes, decode_map: dict[int, bytes]):
text = transformed
matched = True
while matched:
matched = False
for char in text:
if char not in decode_map:
continue
matched = True
text = text.replace(char.to_bytes(), decode_map[char])
return text.decode("utf-8")
# types: misc error: The return type of a generator function should be "Generator" or one of its supertypes
def test() -> None:
yield 3
yield 4
return 5
def run() -> None:
"Run program"
transformed, encode_map = pair_encode("aaabdaaabac")
print((transformed, encode_map))
# types: func-returns-value error: "pair_decode" does not return a value (it only ever returns None)
print(f'{pair_decode(transformed, encode_map) = }')
# types: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
for value in pair_encode_stream("waffles are all the rage in mississippi"):
# types: syntax error: expected an indented block after 'for' statement on line 129
# types: syntax error: expected an indented block after 'for' statement on line 130
if __name__ == "__main__":
print(f"{__title__} v{__version__}\nProgrammed by {__author__}.\n")
run()
And I have no idea if this is relevant, but then apon realizing this is a dmypy bug and copying the program to a different folder and attempting to reproduce this,
the dmypy daemon crashed and apon restarting it and checking the file it told me about the syntax issue.
This is the most recent traceback in log.txt, specified in run arguments.
To be honest not sure if this is related though.
Traceback (most recent call last):
File "mypy/dmypy_server.py", line 236, in serve
File "mypy/dmypy_server.py", line 285, in run_command
File "mypy/dmypy_server.py", line 353, in cmd_run
File "mypy/dmypy_server.py", line 432, in check
File "mypy/dmypy_server.py", line 698, in fine_grained_increment_follow_imports
File "mypy/server/update.py", line 267, in update
File "mypy/server/update.py", line 369, in update_one
File "mypy/server/update.py", line 452, in update_module
File "mypy/server/update.py", line 881, in propagate_changes_using_dependencies
File "mypy/server/update.py", line 1025, in reprocess_nodes
File "mypy/checker.py", line 535, in check_second_pass
File "mypy/checker.py", line 540, in check_partial
File "mypy/checker.py", line 554, in check_top_level
File "mypy/nodes.py", line 1351, in accept
File "mypy/checker.py", line 2934, in visit_assignment_stmt
File "mypy/checker.py", line 2974, in check_type_alias_rvalue
File "mypy/checkexpr.py", line 5832, in accept
File "mypy/errors.py", line 1269, in report_internal_error
File "mypy/checkexpr.py", line 5830, in accept
File "mypy/nodes.py", line 2112, in accept
File "mypy/checkexpr.py", line 3379, in visit_op_expr
File "mypy/checkexpr.py", line 5832, in accept
File "mypy/errors.py", line 1269, in report_internal_error
File "mypy/checkexpr.py", line 5830, in accept
File "mypy/nodes.py", line 2714, in accept
File "mypy/checkexpr.py", line 4730, in visit_type_alias_expr
File "mypy/checkexpr.py", line 4794, in alias_type_in_runtime_context
File "mypy/checker.py", line 6939, in named_generic_type
File "mypy/checker.py", line 6946, in lookup_typeinfo
File "mypy/checker.py", line 7024, in lookup_qualified
KeyError: 'types'
Your Environment
- Mypy version used:
dmypy 1.11.0 - Mypy command-line flags: See above
- Mypy configuration options from
mypy.ini(and other config files): No other configuration files active - Python version used:
Python 3.12.0 (main, Apr 28 2024, 22:42:26) [GCC 13.2.0] on linux
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the dmypy run and suggest behavior with the supplied Python file and command flags, then inspect mypy/dmypy_server.py and the update path in mypy/server/update.py. The traceback also points to checker.py and checkexpr.py. Done means the reported stale status or incorrect suggestion is reproduced and its handling is covered without the daemon crash described here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100