alan-turing-institute / alan-turing-institute/sqlsynthgen

Faster random FK values

Open
#146 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
12
Forks
4
PR merge metrics
No merged PRs in 30d

Description

SSG struggles with big tables from which we have to pick several random foreign key values. `order by random() limit 1` is actually quite slow, I have a 4.8 GB vocabulary table and executing that once takes 20s. In a big schema you might run that a couple of dozen times just during a single pass of `create-data`. There are ways to do this faster that aren't exactly random (not an equal chance for all rows to get picked), and it seems like a somewhat complex topic, so not sure what the right solution here is.

Contributor guide

Open the contributing guide

Research direction

The issue points to the create-data process and its `order by random() limit 1` query for selecting foreign-key values, but names no files or tests. Start by locating that entry point and measuring the query on large tables; done requires selecting and validating a faster approach while documenting its randomness trade-offs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.