[FEA] BSDG: Explore query generation techniques
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
BSDG (Billion Scale Data Generator)
Explore other query generation techniques that allow us to model the relationship between real data and real queries.
Right now we random sample from data and add noise.
In the kNN Decoder model we generate the whole data pool and use heldout queries (from a data distribution aspect, it's still similar to the random sampling above).
Ideally, we would want to
- Figure out the relevant distribution translation (if there is any) between the real data and the real queries.
- And generate our synthetic data and queries to have that same translation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing BSDG's current random-sampling-with-noise approach and the kNN Decoder model described in the issue. Compare how real data and real queries are distributed, then identify a candidate distribution-translation and generation technique. Done should include a documented approach for generating synthetic data and queries with the desired relationship.
Written by the indexing model from the issue text.
Assessment
- Domain
- data, machine-learning, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100