python / python/mypy

Provide better error for wrong ast_serialize version with parallel checking

Open
#21,412 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug parallel checking topic-error-reporting
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Currently if you have ast_serialize==0.2 installed and you try to type check mypy, you get a slightly confusing:

λ python -m mypy --config-file mypy_self_check.ini -p mypy --no-incremental
/Users/shantanu/dev/mypy/mypy/typeshed/stdlib/posixpath.pyi: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 2.1.0+dev.0a72750b6d7f0f2dfaf65f49e63a74882c6b1108
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/shantanu/dev/mypy/mypy/build_worker/__main__.py", line 6, in <module>
    console_entry()
  File "/Users/shantanu/dev/mypy/mypy/build_worker/worker.py", line 356, in console_entry
    main(sys.argv[1:])
  File "/Users/shantanu/dev/mypy/mypy/build_worker/worker.py", line 121, in main
    serve(server, ctx)
  File "/Users/shantanu/dev/mypy/mypy/build_worker/worker.py", line 220, in serve
    load_states(mod_ids, graph, manager, scc_message.import_errors, scc_message.mod_data)
  File "/Users/shantanu/dev/mypy/mypy/build_worker/worker.py", line 284, in load_states
    state.parse_file(raw_data=raw_data)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3225, in parse_file
    with self.wrap_context():
  File "/Users/shantanu/Library/Application Support/uv/python/cpython-3.14.2-macos-aarch64-none/lib/python3.14/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3077, in wrap_context
    yield
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3233, in parse_file
    self.parse_file_inner(source, raw_data)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3197, in parse_file_inner
    self.tree = self.manager.parse_file(
                ~~~~~~~~~~~~~~~~~~~~~~~^
        self.id, self.xpath, source, options=self.options, raw_data=raw_data
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 1294, in parse_file
    tree = load_from_raw(path, id, raw_data, self.errors, options)
  File "/Users/shantanu/dev/mypy/mypy/parse.py", line 80, in load_from_raw
    defs = read_statements(state, data, n)
  File "/Users/shantanu/dev/mypy/mypy/nativeparse.py", line 242, in read_statements
    stmt = read_statement(state, data)
  File "/Users/shantanu/dev/mypy/mypy/nativeparse.py", line 431, in read_statement
    _read_and_set_import_metadata(data, stmt)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/Users/shantanu/dev/mypy/mypy/nativeparse.py", line 2022, in _read_and_set_import_metadata
    flags = read_int(data)
  File "/Users/shantanu/dev/mypy/mypy/cache.py", line 368, in read_int
    assert read_tag(data) == LITERAL_INT
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

/Users/shantanu/dev/mypy/mypy/typeshed/stdlib/posixpath.pyi: note: use --pdb to drop into pdb
error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 2.1.0+dev.0a72750b6d7f0f2dfaf65f49e63a74882c6b1108
Traceback (most recent call last):
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 1392, in receive_worker_message
    return receive(self.workers[idx].conn)
  File "/Users/shantanu/dev/mypy/mypy/ipc.py", line 481, in receive
    raise OSError("No data received")
OSError: No data received

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/shantanu/dev/mypy/mypy/__main__.py", line 52, in <module>
    console_entry()
  File "/Users/shantanu/dev/mypy/mypy/__main__.py", line 16, in console_entry
    main()
    ~~~~^^
  File "/Users/shantanu/dev/mypy/mypy/main.py", line 154, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
                              ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shantanu/dev/mypy/mypy/main.py", line 244, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 422, in build
    result = build_inner(
        sources,
    ...<9 lines>...
        metastore,
    )
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 537, in build_inner
    graph = dispatch(sources, manager, stdout, connect_threads)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 4150, in dispatch
    process_graph(graph, manager)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 4618, in process_graph
    done, still_working, results = manager.wait_for_done(graph)
                                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 1488, in wait_for_done
    return self.wait_for_done_workers(graph)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 1507, in wait_for_done_workers
    buf = self.receive_worker_message(idx)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 1400, in receive_worker_message
    raise OSError(
        f"Worker {idx} disconnected before sending data ({exit_status})"
    ) from exc
OSError: Worker 0 disconnected before sending data (exit code 2)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the failing path in mypy/nativeparse.py and mypy/cache.py, then trace how mypy/build_worker/worker.py handles the corrupted or incompatible serialized data during parallel checking. Reproduce the ast_serialize==0.2 failure using the command in the issue. Done means an incompatible version produces a clear user-facing error instead of an assertion and worker-disconnect traceback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.