apache / apache/arrow-cookbook
[Python][Flight] pyarrow.flight.FlightServerBase use threading instead of multiprocessing?
- Dominant language
- C++
- Stars
- 108
- Forks
- 49
- Avg merge
- 2h 14m
- Merged PRs (30d)
- 1
Description
I tried to use arrow flight, but when testing the performance (`do_exchange`), I found that the cpu usage can only be around 100%, maybe it is the GIL problem of python multi-threading.
I didn't find anything about setting the thread pool for `FlightServerBase`, so it seems that if you want to improve performance, it seems that you can only use `multiprocessing.Pool` internally to handle the internal working logic.
Do you have any other suggestions? Thanks!
Contributor guide
Research direction
Start by locating the Python FlightServerBase implementation and the do_exchange entry point; the issue does not name files or tests. Reproduce the reported performance behavior, then review the comment discussion for the proposed concurrency direction. Done would require an agreed threading or multiprocessing design and benchmark evidence, but the issue currently asks for suggestions rather than defining a change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100