block / block/elasticgraph

Prune orphaned __nested_sourced_data entries to avoid unbounded growth

Open
#1,270 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
88
Forks
36
Avg merge
5d 10h
Merged PRs (30d)
18

Description

## Background

`sourced_from` lets a field on an indexed type be populated from a *different* source type's events. https://github.com/block/elasticgraph/pull/1252 extended this to fields on **nested elements** of a document.

To make that work, the indexing script (`index_data.painless`) buffers each related event's nested sourced fields in a hidden `__nested_sourced_data` map, and re-applies the buffer on every event (so a later event overwriting the nested array doesn't drop the sourced fields).

## Problem

Nothing ever removes entries from `__nested_sourced_data`. When a nested element is dropped from its parent event, its buffer entry is orphaned — harmlessly skipped on apply, but never reclaimed. For documents whose nested collections churn, the buffer grows without bound and adds work to every update.

## Proposal

Reclaim orphaned entries from `__nested_sourced_data`. The hard part is that the obvious approach — prune whenever the target element can't be found — does **not** work: a missing element is ambiguous between "permanently removed" and "not yet arrived" (out-of-order ingestion, the case the buffer exists to support). A naive prune would delete legitimately-pending entries — in fact on the same event that stored them, since storing is immediately followed by re-application.

A correct fix needs a way to distinguish removal from not-yet-arrived.

Contributor guide

Open the contributing guide

Research direction

Start with the indexing script, index_data.painless, and trace how __nested_sourced_data is stored and reapplied for nested elements. Determine how removal can be distinguished from out-of-order arrival, then verify that orphaned entries are reclaimed without deleting legitimately pending data.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
databases, search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.