trinodb / trinodb/trino-python-client
benchmark trino client's speed of retrieving data, it seems the bottleneck of the data pipeline
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 439
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
Expected behavior
when using simple select * from db, the speed should not be less than the original database's speed, otherwise the system overall will be delayed by trino itself.
for example, when getting data directly from database with sqlalchemy, the speed would reach 100MB/s, while when trino is getting in, the speed overall decreases to only 10MB/s.
Actual behavior
the speed of trino should be no less than the database one.
Steps To Reproduce
I have tested the bottleneck of python client.
I used a memory connector which means the data is reside in the trino itself, the time records only for data getting out of the trino and get in to the client.
however, this bottleneck is only about 10-20MB/s, while my backend database could get about 100MB/s in a single connection.
Log output
Operating System
ubuntu 20.04
Trino Python client version
lastest
Trino Server version
lastest
Python version
3.10
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported throughput difference with the memory connector, Python 3.10, and the provided SQLAlchemy comparison. Measure data retrieval from the Trino client separately from backend database access, then identify and document the client-side bottleneck and verify the results against the reported 10–20MB/s and 100MB/s rates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100