replikativ / replikativ/datahike-serverless
A GC custodian — required, and a serverless deployment has nowhere to put one
Nobody has claimed this yet.
- Dominant language
- Clojure
- Stars
- 1
- Forks
- 1
- Avg merge
- 6h 35m
- Merged PRs (30d)
- 3
Description
Datahike's storage GC (d/gc-storage) must run in a writer JVM. A fully
serverless deployment has nowhere to run one, so today nothing here collects
garbage and the bucket only grows.
This is not an optimisation. Measured on one tenant in the source prototype: 197
objects of which 40 were live after GC — 80% garbage. A third-party production
bucket was found at 16x its live state for want of a custodian.
Constraints that make it non-trivial:
- It has to hold the writer role, and writers are single-writer until #878 —
so the custodian and the writer function must not be live at the same time on
the same tenant, or the custodian must be the writer. - It is per tenant, and the number of tenants is the whole point of the model,
so it needs a schedule/queue rather than a loop over everything. - It wants to run when the tenant is idle, which is most of the time in the
regime where serverless wins at all.
Options to evaluate, cheapest first:
- A scheduled long-lived container (ECS/Fargate task, Cloud Run job) holding the
writer role, walking a work queue of tenants — the boring answer, and it
removes the "no always-on process" property that some of the economics rest on.
Quantify what that costs. - Fold GC into the writer function itself, opportunistically, after a commit,
with a budget. Attractive because the writer is already the only writer;
risky because a function has an execution deadline and GC does not. - Wait for #878 and let any writer safely GC under CAS.
Whatever is chosen, the README's "Not built, and required" line should become a
description of what actually runs.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the README's “Not built, and required” line and the d/gc-storage behavior described in the issue. Evaluate the scheduled ECS/Fargate option, writer-function integration, and waiting for #878, including the requested cost and concurrency constraints. Done means selecting and documenting what runs per tenant and updating the README accordingly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, clojure
- Domain
- backend, cloud, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100