Support sourcing list fields onto a nested element (with tracked __counts)
- Dominant language
- Ruby
- Stars
- 88
- Forks
- 36
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 18
Description
## Background
ElasticGraph's `sourced_from` feature lets a field on an indexed type be populated from a *different* source type's events, rather than from the indexed type's own events. https://github.com/block/elasticgraph/pull/1252 extended this so a `sourced_from` field can live on a **nested element** of an indexed document, not just at the top level of the document.
Separately, ElasticGraph maintains a hidden `__counts` map on indexed documents to track the sizes of list fields. When a list field's contents change, ElasticGraph recomputes its count so that list-size-based queries stay correct.
## Feature gap
Nested `sourced_from` currently supports sourcing **scalar** and **object** fields onto a nested element, but not **list** fields. In order to support sourcing nested **list** fields, the `__counts` tracking must work for these fields (there is currently no mechanism for this for the nested sourced field case).
## Interim state
Until this is supported, a schema-dump-time validation will reject defining such a schema with a clear error pointing to this issue.
Contributor guide
Research direction
Start by tracing the existing nested `sourced_from` handling and the hidden `__counts` tracking for list fields. Review the schema-dump-time validation that currently rejects this case; done means nested sourced list fields are supported with correct counts and are no longer rejected by that validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, ruby
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100