elastic / elastic/roadmap

Columnar Logs (Tech Preview)

Open
#291 0 comments 0 reactions 1 assignee Claimed by @yannis-roussos View on GitHub
Component: Elasticsearch product-area:observability product-area:search product-area:security v9.5.0
Dominant language
No language data
Stars
6
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Value Prop

**Columnar Logs** is the first specialized profile of [**Columnar Mode**](https://github.com/elastic/roadmap/issues/290), focused on logs. It adds logging-oriented defaults on top of the columnar foundation and stores log data once, in the column store (doc values) — a columnar evolution of LogsDB with TSDS-style optimizations — so logs take meaningfully less storage than they do today.

The key difference from pure Columnar Mode: **Columnar Logs keeps a single inverted index on the `message` field**, because fast free-text search on log messages is critical. Every other field goes fully columnar and is not indexed by default. The result is fast log search with a much smaller storage footprint.

Because some of these optimizations are breaking, Columnar Logs is a **new, opt-in index mode** rather than a change to LogsDB. Existing LogsDB indices are untouched; users adopt Columnar Logs where it fits.

## Expected Outcome

The Tech Preview delivers the fundamental building blocks. As with Columnar Mode, it front-loads every change that would be breaking if introduced later.

New index mode (`columnar-logsdb`), with:
- Log data stored once as doc values (columnar), inheriting the Columnar Mode building blocks
- A single inverted index on the `message` field (and any other fields defined as text either explicitly or by integrations) for fast free-text log search
- All other fields columnar and not indexed by default
- Adds logging-oriented defaults: a default `@timestamp` mapping, and index sorting on `host.name` and `@timestamp` when those fields are present.

GA will add performance optimizations and non-breaking features on top of this foundation.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.