block / block/elasticgraph

Support non-indexed sourced_from source types in data warehouse ingestion

Open
#1,359 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
88
Forks
36
Avg merge
5d 10h
Merged PRs (30d)
18

Description

## Context

#1273 added support for pure-source types: types that feed `sourced_from` fields on other types without calling `t.index` (see #1342, #1343, #1346, #1347, #1356). Events for these types are validated and their sourced fields flow to the destination type's index, but the warehouse gems don't handle them.

## Problem

Events for a non-indexed source type never reach the data warehouse:

1. **No warehouse table is defined.** Warehouse tables hang entirely off indices: every `t.index` automatically gets a table (`FactoryExtension#new_index` in `elasticgraph-warehouse/lib/elastic_graph/warehouse/schema_definition/factory_extension.rb`), the `warehouse_table` / `exclude_from_warehouse` customization API lives on the index object (`index_extension.rb`), `WarehouseTable` itself is `Data.define(:name, :index)`, and `ResultsExtension#generate_warehouse_config` collects tables via each type's `own_index_def`. A type with no index has no table and no place to declare one.
2. **No operation survives to be dumped.** `WarehouseDumper#build_jsonl_file_from` treats the `__self` update target (where `op.update_target.type == op.event.fetch("type")`) as the event's "primary write" (`elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb:116`). A non-indexed type has no `__self` update target — its events only produce operations targeting the *destination* type's index — so every operation is filtered out and no S3 file is written (the dumper skips empty groups).

Note that (2) isn't just a matter of relaxing the filter: no operation carries the full source-event payload (the `sourced_from` update targets' script params only contain the sourced field values, shaped for the destination document). Warehouse ingestion would need a representation of the raw source event itself.

The warehouse is arguably the one place raw source events *should* remain visible — the datastore only ever holds the copied fields — so silently dropping them seems wrong.

## What's needed

- Generalize warehouse table generation to cover ingestible non-indexed types (the same "ingestible types" set used for the event envelope enum in #1346, exposed as `Results#sourced_update_targets_by_source_type_name`). The column derivation should port cleanly — `WarehouseTable#table_schema` already builds from `indexing_fields_by_name_in_index`, which non-indexed types have — but the table needs to be keyed off the type rather than an index, and the `warehouse_table` / `exclude_from_warehouse` API needs a home for types without an index block.
- Give `WarehouseDumper` a primary-write signal (and a payload source) that doesn't depend on a `__self` update target.

## Open question

Indexed types get a warehouse table automatically, with `exclude_from_warehouse` as the opt-out. Should pure-source types follow the same automatic-with-opt-out convention, or should their tables be explicit opt-in?

Contributor guide

Open the contributing guide

Research direction

Start with Results#sourced_update_targets_by_source_type_name, WarehouseTable#table_schema, and the named FactoryExtension, index_extension, ResultsExtension, and WarehouseDumper files. Trace how non-indexed source types reach configuration and how build_jsonl_file_from selects primary writes; done means their tables and raw source events are represented in warehouse output, with the opt-in or opt-out convention resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, ruby
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.