alan-turing-institute / alan-turing-institute/sqlsynthgen
Faster random FK values
- Dominant language
- Python
- Stars
- 12
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
SSG struggles with big tables from which we have to pick several random foreign key values. `order by random() limit 1` is actually quite slow, I have a 4.8 GB vocabulary table and executing that once takes 20s. In a big schema you might run that a couple of dozen times just during a single pass of `create-data`. There are ways to do this faster that aren't exactly random (not an equal chance for all rows to get picked), and it seems like a somewhat complex topic, so not sure what the right solution here is.
Contributor guide
Research direction
The issue points to the create-data process and its `order by random() limit 1` query for selecting foreign-key values, but names no files or tests. Start by locating that entry point and measuring the query on large tables; done requires selecting and validating a faster approach while documenting its randomness trade-offs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100