elastic / elastic/elastic-package

testing: add LogsDB Columnar mode to enable wide adoption testing

Open
#3,700 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

## Background

LogsDB Columnar is a new Elasticsearch storage format currently in tech review,
targeting the 9.6 feature freeze. It must be explicitly enabled per data stream
via the `logs@custom` component template (to apply broadly) or a more specific
`logs-.@custom` component template (to target a single
data stream). Unlike standard LogsDB, it cannot be applied automatically.

The following are breaking semantic changes in the current tech preview state —
these may evolve as the feature matures:

- ~Flat mappings enforced — nested field type is completely unsupported~
- No original source stored; unmapped fields under `dynamic: false` are silently
discarded rather than preserved in `_source`/`_ignored_source`
- Runtime fields and `doc_values: false` are unsupported
- New mapping attributes: `multi_value`, `nullability`

The Elasticsearch team's goal is for LogsDB Columnar mode to become the default
for all logs data streams in integrations.

## Request

Add support to `elastic-package` to run system tests (which actually index data)
with LogsDB Columnar mode enabled on `logs-*` data streams, gated behind the
LogsDB Columnar feature flag availability (named `columnar_index_mode`).

Concretely:

- A flag (e.g. `--logsdb-columnar` or a named profile) that injects the
necessary component template settings to enable LogsDB Columnar mode on logs
data streams during system tests
- Works alongside the existing `--logsdb` flag / profile
- No behavior change for tests that don't opt in

## Goals

1. **Surface ES indexing errors early** — nested field conflicts, unsupported
mapping attributes, dynamic false data-loss scenarios
2. **Surface test assertion failures** — cases where system test data checks
break under LogsDB Columnar mode's different source/query semantics
3. **Enable a wide ecosystem sweep** — once the feature flag is available,
integration teams can run this broadly to triage adoption readiness

## Known blocker: nested fields

Any data stream whose mapping uses the `nested` field type will fail under
LogsDB Columnar mode — nested is fundamentally incompatible with flat mappings.
The test runner should either skip LogsDB Columnar mode for those data streams
or collect the errors separately so they don't mask unrelated failures. Roughly
4% of logs data streams across integrations are known to use nested fields.

## Context

This came out of the Logs Program working group (2026-06-24). The LogsDB team
is looking for help validating the feature while it is in tech preview.

PR to add docs for columnar: https://github.com/elastic/elasticsearch/pull/151739

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the existing --logsdb flag or profile and the system-test component-template injection for logs-* data streams. Add an opt-in path gated by the columnar_index_mode feature flag, preserving behavior without it and alongside --logsdb. Verify indexing and assertions under columnar mode, and confirm nested-field data streams are skipped or reported separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, testing, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.