apache / apache/arrow-cookbook

[Python][Flight] pyarrow.flight.FlightServerBase use threading instead of multiprocessing?

Open
#236 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
108
Forks
49
Avg merge
2h 14m
Merged PRs (30d)
1

Description

I tried to use arrow flight, but when testing the performance (`do_exchange`), I found that the cpu usage can only be around 100%, maybe it is the GIL problem of python multi-threading.

I didn't find anything about setting the thread pool for `FlightServerBase`, so it seems that if you want to improve performance, it seems that you can only use `multiprocessing.Pool` internally to handle the internal working logic.

Do you have any other suggestions? Thanks!

Contributor guide

Open the contributing guide

Research direction

Start by locating the Python FlightServerBase implementation and the do_exchange entry point; the issue does not name files or tests. Reproduce the reported performance behavior, then review the comment discussion for the proposed concurrency direction. Done would require an agreed threading or multiprocessing design and benchmark evidence, but the issue currently asks for suggestions rather than defining a change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.