apache / apache/accumulo

Consider/experiment with avoiding scan server refs by delaying GC

Open
#4,538 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

Related to #4529, which proposed moving scan server file refs to their own table.

An alternative to consider would be to try to avoid the need to create scan server refs in the first place. This could be done by ensuring GC candidates are not deleted immediately, but deletion is delayed enough to give time for existing scan servers to finish using them (in other words, delay at least as long as it takes for the scan server file ref cache to expire).

One advantage of this would be to reduce the use of the metadata table and the remove the need to constantly update the file refs based on the current scan servers.

Even delaying GC, some refs may still be needed for long-running scans that are not yet complete.

So, some challenges include:

* tracking how long files need to wait before they are deleted (if the GC keeps track in its memory, it could use a lot of memory; if it's persisted, then we have to have a way of representing the time that is recorded, and global time tracking in a cluster can be tricky)
* determining when a scan server needs to store a ref for a long-running scan

I'm not certain this is a better idea than #4528, but there may be advantages that are worth pursuing instead of that (or maybe in addition to that).

Contributor guide

Open the contributing guide

Research direction

Start by reading related issues #4529 and #4528, then inspect the existing garbage-collection and scan-server reference behavior described here. The issue does not name files or tests, and completion would first require deciding whether delayed GC is preferable and defining how its timing and long-running scans should be handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.