trinodb / trinodb/trino-python-client

benchmark trino client's speed of retrieving data, it seems the bottleneck of the data pipeline

Open
#404 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
439
Forks
206
PR merge metrics
No merged PRs in 30d

Description

Expected behavior

when using simple select * from db, the speed should not be less than the original database's speed, otherwise the system overall will be delayed by trino itself.

for example, when getting data directly from database with sqlalchemy, the speed would reach 100MB/s, while when trino is getting in, the speed overall decreases to only 10MB/s.

Actual behavior

the speed of trino should be no less than the database one.

Steps To Reproduce

I have tested the bottleneck of python client.

I used a memory connector which means the data is reside in the trino itself, the time records only for data getting out of the trino and get in to the client.

however, this bottleneck is only about 10-20MB/s, while my backend database could get about 100MB/s in a single connection.

Log output
截屏2023-08-24 13 42 05
Operating System

ubuntu 20.04

Trino Python client version

lastest

Trino Server version

lastest

Python version

3.10

Are you willing to submit PR?
  • Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported throughput difference with the memory connector, Python 3.10, and the provided SQLAlchemy comparison. Measure data retrieval from the Trino client separately from backend database access, then identify and document the client-side bottleneck and verify the results against the reported 10–20MB/s and 100MB/s rates.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.