MaibornWolff / MaibornWolff/database-performance-comparison
Analyze/improve query performance for Yugabyte
- Dominant language
- Python
- Stars
- 28
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
With the newest version of YugabyteDB and some optimizations insert performance for YSQL has increased dramatically to about 295000 inserts/s in our testcase (copy mode, batch size 1000), bringing it up to par with the YCQL interface (which also increased slightly to about 290000).
But the query performance is unexpectedly slow. For the usecase with 500 million rows I get the following query times for our test queries:
* count-events: 160s (slower than other PostgreSQL-compatible databases but not terribly so)
* temperature-min-max: 0.03s (this is good as it shows the index is used as we want it)
* temperature-stats: 1530s (compared to other PostgreSQL-compatible databases this is an order of magnitude slower)
* temperature-stats-per-device: 1717s (again an order of magnitude slower)
* newest-per-device: This fails with `psycopg2.errors.ConfigurationLimitExceeded: temporary file size exceeds temp_file_limit (1048576kB)`
One more issue: When creating the index `events_device_ts` sometimes the following error happens: `psycopg2.errors.InternalError_: Aborted: ERROR: Query error: schema version mismatch for table 000033e6000030008000000000004104: expected 1, got 0`. The index still seems to get created as running the `CREATE INDEX` statement again yields `NOTICE: relation "events_device_ts" already exists, skipping`.
For reference the commands:
* `helm install yugabyte yugabytedb/yugabyte -f dbinstall/yugabyte-values.yaml --version 2.15.0`
* `python run.py insert --target yugabyte_sql -w 16 -r 1 --num-inserts 31250000 --batch 1000 --primary-key sql`
* `python run.py query --target yugabyte_sql -w 1 -r 1 --extra-option create_indices=true`
@FranckPachot I hope your offer of help is still valid? I would be grateful for some pointers on how to deal with the `temp_file_limit` and any ideas on how to speed up the other queries. With the latest optimizations insert performance is formidable, so I would love to get query performance to match.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing run.py and dbinstall/yugabyte-values.yaml, then reproduce the insert and query commands against YugabyteDB 2.15.0. Compare the listed query timings, the temp_file_limit failure, and the events_device_ts schema-version error. Done means identifying the causes and recording reproducible findings or validated performance improvements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, postgresql, python
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100