questdb / questdb/py-questdb-client

Concurrent first `import pandas` from worker threads segfaults the interpreter

Open
#144 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
71
Forks
14
Avg merge
1h 7m
Merged PRs (30d)
1

Description

src/questdb/egress.pxi:1886, :2189 and nine other lazy-import sites do
import pandas / import numpy on every call — that is, on whichever thread
calls to_pandas() first. Eight threads doing that as the process's first
pandas import kill the interpreter: exit 139, reproducible every run,
mi_thread_init in libarrow's mimalloc dereferencing NULL under
pandas._libs.lib.maybe_convert_objects.

This is not a defect in this client. The same crash reproduces with no
questdb involvement at all, from eight threads whose first statement is
import pandas plus any object-str column. It is the pandas 3.0.3 /
pyarrow 25.0.0 / CPython 3.14 combination, and only a str-bearing column
triggers it — SELECT l, ts, decimals and UUIDs all survive. Pre-importing
pandas on the main thread makes it pass.

What makes it worth recording here is that this client's threading guarantee
(CHANGELOG.rst:182-185, and the QueryResult hand-off docstring) points
users directly at the pattern that triggers it.

Mitigation. A module-level cached import behind a lock — the shape
_dataframe_may_import_deps() already uses — would keep the first import on
one thread. Worth a line in the threading docs either way.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the lazy-import sites in src/questdb/egress.pxi, especially lines 1886 and 2189, and compare them with _dataframe_may_import_deps(). Read the threading guarantee in CHANGELOG.rst:182-185 and the QueryResult hand-off docstring, then reproduce the eight-thread first-import case. Done means the issue’s chosen mitigation or documentation clearly addresses the documented threading pattern without claiming the upstream pandas crash is fixed.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
api, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.