Consider/experiment with avoiding scan server refs by delaying GC
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
Related to #4529, which proposed moving scan server file refs to their own table.
An alternative to consider would be to try to avoid the need to create scan server refs in the first place. This could be done by ensuring GC candidates are not deleted immediately, but deletion is delayed enough to give time for existing scan servers to finish using them (in other words, delay at least as long as it takes for the scan server file ref cache to expire).
One advantage of this would be to reduce the use of the metadata table and the remove the need to constantly update the file refs based on the current scan servers.
Even delaying GC, some refs may still be needed for long-running scans that are not yet complete.
So, some challenges include:
* tracking how long files need to wait before they are deleted (if the GC keeps track in its memory, it could use a lot of memory; if it's persisted, then we have to have a way of representing the time that is recorded, and global time tracking in a cluster can be tricky)
* determining when a scan server needs to store a ref for a long-running scan
I'm not certain this is a better idea than #4528, but there may be advantages that are worth pursuing instead of that (or maybe in addition to that).
Contributor guide
Research direction
Start by reading related issues #4529 and #4528, then inspect the existing garbage-collection and scan-server reference behavior described here. The issue does not name files or tests, and completion would first require deciding whether delayed GC is preferable and defining how its timing and long-running scans should be handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100