Allow data reuse when extending a dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 314
- Forks
- 77
- Avg merge
- 4d 6m
- Merged PRs (30d)
- 10
Description
Using a DatasetBuilder to extend an already-assembled Dataset should be able to allow the initial data set's storage to be reused.
Right now, the code does not do that very well, because MatrixRelationshipSet._init_structures calls combine_chunks on the table. If my memory is correct, this was to simplify some Rust optimizations.
To improve memory use, we should remove that combine_chunks call and fix whatever breaks to enable data reuse.
We should also add more tests using data set extension to make sure the code is well-exercised.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating MatrixRelationshipSet._init_structures and the DatasetBuilder path used to extend an assembled Dataset. Trace the combine_chunks call and how the initial dataset's storage is passed through, then run the relevant tests while adding coverage for dataset extension. Done means extension can reuse the existing storage without breaking behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100