Prune orphaned __nested_sourced_data entries to avoid unbounded growth
- Dominant language
- Ruby
- Stars
- 88
- Forks
- 36
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 18
Description
## Background
`sourced_from` lets a field on an indexed type be populated from a *different* source type's events. https://github.com/block/elasticgraph/pull/1252 extended this to fields on **nested elements** of a document.
To make that work, the indexing script (`index_data.painless`) buffers each related event's nested sourced fields in a hidden `__nested_sourced_data` map, and re-applies the buffer on every event (so a later event overwriting the nested array doesn't drop the sourced fields).
## Problem
Nothing ever removes entries from `__nested_sourced_data`. When a nested element is dropped from its parent event, its buffer entry is orphaned — harmlessly skipped on apply, but never reclaimed. For documents whose nested collections churn, the buffer grows without bound and adds work to every update.
## Proposal
Reclaim orphaned entries from `__nested_sourced_data`. The hard part is that the obvious approach — prune whenever the target element can't be found — does **not** work: a missing element is ambiguous between "permanently removed" and "not yet arrived" (out-of-order ingestion, the case the buffer exists to support). A naive prune would delete legitimately-pending entries — in fact on the same event that stored them, since storing is immediately followed by re-application.
A correct fix needs a way to distinguish removal from not-yet-arrived.
Contributor guide
Research direction
Start with the indexing script, index_data.painless, and trace how __nested_sourced_data is stored and reapplied for nested elements. Determine how removal can be distinguished from out-of-order arrival, then verify that orphaned entries are reclaimed without deleting legitimately pending data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- databases, search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100