apache / apache/arrow-rs

Arrow Flight Performance -- Rust vs Python (C++)

Open
#6,670 13 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Which part is this question about**
Arrow flight, FlightDataEncoderBuilder, do_get

**Describe your question**
Is it expected that Arrow's Python (C++) Flight implementation encodes data more efficiently than arrow-rs?

**Additional context**
Hello.
After discussion with @alamb, I am filing an issue here.

Unsure if this is a bug, or if it's expected, or if there's just an issue with my code, but after running some tests, it seems that Rust's encoding takes more time and resources than Python.

I am running two servers, one in Python and the other in Rust, with the same simple design:
-Create a `Table`/`RecordBatch` before starting the flight service, which the service will hold in memory when running.
-When receiving a request (in `do_get`), simply provide a view of the data to `fl.RecordBatchStream` in Python / `FlightDataEncoderBuilder` in Rust.

Because nothing is really happening on the Python side (just providing a view to a `Table`), and a single request is not holding the GIL for a significant amount of time, I imagine I'm ultimately measuring the C++ Arrow Flight implementation.

I have run two tests:
1. Python script which sends *n* requests sequentially to each server, consuming the entire stream (`flightclient.do_get().read_all()`) and displays the average response time for each server.
2. Using the Locust framework, load testing the maximum RPS capabilities of the servers (used `taskset -c` to seperate locust users and server).

I observe the following from the tests:
1. As the size of data sent to the client increases, the difference of average response time between Python and Rust servers also increases (in favor of Python server).
2. Similarily, as the amount of data increases, Python is able to achieve a higher RPS than Rust.

The Rust server's CPUs are fully utilized (using more than Python server in certain cases). After profiling with `perf`, I am seeing a lot of CPU usage related to memory movement.

You can access my code here: https://github.com/vichry2/flight-benchmark

Thank you for your help!

Contributor guide

Open the contributing guide

Research direction

Start with the linked flight-benchmark and the reported perf results, focusing on FlightDataEncoderBuilder and do_get. Compare the Rust server with the Python FlightDataEncoder path under the described sequential and Locust tests. Done means establishing whether the encoding and memory-movement difference is expected or identifying a reproducible Rust performance issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python, rust
Domain
api, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.