msgspec / msgspec/msgspec

Segmentation fault when using msgspec.Meta

Open
#717 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.1k
Forks
193
Avg merge
3d 20h
Merged PRs (30d)
15

Description

Description

Hey there,

First of all, thanks for this very nice library.

I wanted to report an issue, that seems to be related to msgspec, but you will be able to better assess whether that is indeed the case.

Our CI pipelines in GitHub started failing randomly (say, 20% failure rate), with a segmentation fault error. The failures disappeared after rerunning the failing jobs (one or more times depending on our luck). The failure was when running pytest (using python -m pytest). After setting python execution to a verbose mode, we noticed that, when it failed, then it always failed when destroying the following module: pysdmx.model.types. This module uses msgspec.Meta.

(many more lines above)
# destroy pysdmx.model.types

Fatal Python error: Segmentation fault
2024-08-06T14:19:01.4875904Z 
2024-08-06T14:19:01.4876092Z Current thread 0x00007f71d8383000 (most recent call first):
2024-08-06T14:19:01.4876697Z   Garbage-collecting
2024-08-06T14:19:01.4876943Z   <no Python frame>
2024-08-06T14:19:02.9130965Z /home/runner/work/_temp/7afba60b-b9e9-4f16-9741-aa8c0d7ce98d.sh: line 1:  1920 Segmentation fault      (core dumped) poetry run python -X faulthandler -v -m pytest -p no:faulthandler --full-trace tests/api/fmr

This module was very simple:

from typing import Annotated

from msgspec import Meta

NC_NAME_ID_TYPE = Annotated[str, Meta(pattern=r"^[A-Za-z][A-Za-z\d_-]*$")]

Once we removed this, our CI pipelines started working again, 100% of the time. So, our suspicion is that there might be cases when using msgspec.Meta prevents created objects from being destroyed, but this is just a guess based on the log above and the fact that the error did not occur again when we stopped using that class. However, it could also be that our suspicion is wrong. In which case, feel free to close this issue :-).

Thanks,
Xavier

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported module-destruction failure with the Annotated[str, Meta(...)] example and the python -X faulthandler -v -m pytest command. Focus on the pysdmx.model.types destruction point and garbage collection behavior; the issue is done when the intermittent segmentation fault can be explained and no longer occurs in the relevant test run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.