block / block/elasticgraph

Support sourcing list fields onto a nested element (with tracked __counts)

Open
#1,266 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
88
Forks
36
Avg merge
5d 10h
Merged PRs (30d)
18

Description

## Background

ElasticGraph's `sourced_from` feature lets a field on an indexed type be populated from a *different* source type's events, rather than from the indexed type's own events. https://github.com/block/elasticgraph/pull/1252 extended this so a `sourced_from` field can live on a **nested element** of an indexed document, not just at the top level of the document.

Separately, ElasticGraph maintains a hidden `__counts` map on indexed documents to track the sizes of list fields. When a list field's contents change, ElasticGraph recomputes its count so that list-size-based queries stay correct.

## Feature gap

Nested `sourced_from` currently supports sourcing **scalar** and **object** fields onto a nested element, but not **list** fields. In order to support sourcing nested **list** fields, the `__counts` tracking must work for these fields (there is currently no mechanism for this for the nested sourced field case).

## Interim state

Until this is supported, a schema-dump-time validation will reject defining such a schema with a clear error pointing to this issue.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing nested `sourced_from` handling and the hidden `__counts` tracking for list fields. Review the schema-dump-time validation that currently rejects this case; done means nested sourced list fields are supported with correct counts and are no longer rejected by that validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, ruby
Domain
backend, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.