lance-format / lance-format/lance-ray
support for distributed clean up old versions
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 41
- Forks
- 48
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 5
Description
Currently
ds = lance.dataset("s3://test/xxxxx.lance", storage_options=storage_options) ds.cleanup_old_versions( older_than=timedelta(seconds=10), delete_unverified=True, error_if_tagged_old_versions=False )
Could take a long time when thousand billions without any log print. Raise an issue to track any distributed clean way.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Python cleanup_old_versions call shown in the issue and how this repository integrates Lance with Ray. Investigate the existing cleanup path and determine how distributed work should be coordinated for very large datasets. Done should provide a defined distributed cleanup approach that avoids a long silent operation, with tests or observable progress covering the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100