MaibornWolff / MaibornWolff/database-performance-comparison

Analyze/improve query performance for Yugabyte

Open
#7 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
10
PR merge metrics
No merged PRs in 30d

Description

With the newest version of YugabyteDB and some optimizations insert performance for YSQL has increased dramatically to about 295000 inserts/s in our testcase (copy mode, batch size 1000), bringing it up to par with the YCQL interface (which also increased slightly to about 290000).

But the query performance is unexpectedly slow. For the usecase with 500 million rows I get the following query times for our test queries:
* count-events: 160s (slower than other PostgreSQL-compatible databases but not terribly so)
* temperature-min-max: 0.03s (this is good as it shows the index is used as we want it)
* temperature-stats: 1530s (compared to other PostgreSQL-compatible databases this is an order of magnitude slower)
* temperature-stats-per-device: 1717s (again an order of magnitude slower)
* newest-per-device: This fails with `psycopg2.errors.ConfigurationLimitExceeded: temporary file size exceeds temp_file_limit (1048576kB)`

One more issue: When creating the index `events_device_ts` sometimes the following error happens: `psycopg2.errors.InternalError_: Aborted: ERROR: Query error: schema version mismatch for table 000033e6000030008000000000004104: expected 1, got 0`. The index still seems to get created as running the `CREATE INDEX` statement again yields `NOTICE: relation "events_device_ts" already exists, skipping`.

For reference the commands:
* `helm install yugabyte yugabytedb/yugabyte -f dbinstall/yugabyte-values.yaml --version 2.15.0`
* `python run.py insert --target yugabyte_sql -w 16 -r 1 --num-inserts 31250000 --batch 1000 --primary-key sql`
* `python run.py query --target yugabyte_sql -w 1 -r 1 --extra-option create_indices=true`

@FranckPachot I hope your offer of help is still valid? I would be grateful for some pointers on how to deal with the `temp_file_limit` and any ideas on how to speed up the other queries. With the latest optimizations insert performance is formidable, so I would love to get query performance to match.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing run.py and dbinstall/yugabyte-values.yaml, then reproduce the insert and query commands against YugabyteDB 2.15.0. Compare the listed query timings, the temp_file_limit failure, and the events_device_ts schema-version error. Done means identifying the causes and recording reproducible findings or validated performance improvements.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, postgresql, python
Domain
databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.