opensearch-project / opensearch-project/sql

[FEATURE] Enhance `timechart` PPL Command

Open
#4,259 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

calcite enhancement PPL
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Enhance timechart PPL Command

Summary

Extend the existing timechart command in OpenSearch PPL with advanced parameters including bins, usenull, and cont to provide comprehensive time-series visualization capabilities. This enhancement builds upon the initial timechart implementation to support dynamic binning, extensive null value handling, and time gap filling.

Background

The initial timechart RFC implemented core functionality with span-based time binning and series management. Several advanced features commonly used in time-series analysis were deferred for future implementation.

Current timechart limitations:

  • Only supports span-based binning (fixed time intervals)
  • No automatic bin count calculation
  • Missing time periods not handled consistently
  • Null values always appear as separate series
  • No control over time gap filling behavior

Advanced capabilities needed:

  • Dynamic bin calculation based on data range
  • Flexible null value handling
  • Automatic time gap filling with configurable behavior

Proposed Implementation

Complete Timechart Command Syntax with New Parameters
... 
| timechart 
    [bin-options] 
    [limit=<int>]
    [useother=<boolean>]
    [usenull=<boolean>]
    [cont=<boolean>] 
    (<multiple-aggregation-functions> [by <field>])

Detailed Feature Requirements

1. Time binning
bins=int - Dynamic equal-width time binning

source=logs | timechart bins=50 count() by status
source=logs | timechart bins=100 avg(response_time)

  • Automatically calculates optimal time intervals based on data time range
  • Divides total time span into specified number of equal buckets
  • Default: bins=100 when neither span nor bins specified
  • Priority: span overrides bins if both specified
  • Bin command is already implemented. Timechart can call visitBin to support all bin options.
2. Gap Filling and Continuity
usenull=<boolean> - Null value handling

source=logs | timechart span=1h count by status usenull=false
Default: usenull=true
Controls whether null values appear as separate series

  • usenull=true behavior is already implemented by default. Null values will always be considered a separate category and not included in the limit in the current behavior.
cont=<boolean> - Fill time gaps

source=logs | timechart span=1h count cont=true
Default: cont=true
Automatically fills missing time periods with null values
Essential for consistent time-series visualization

3. Multiple Aggregation Functions

source=events | timechart count avg(cpu_usage)
source=events | timechart count, avg(cpu_usage) by host

  • Accept comma-separated and not comma-separated multiple aggregation functions.
  • Each aggregation produces its own column for visualization.
  • Works with all current timechart parameters (span, bins, limit, useother, usenull, cont).
  • Bin calculation, null handling, and gap filling are applied consistently across all series.

Implementation Details

  • Bin functionality fully implemented - visitBin(), BinUtils, BinHandlerFactory with all bin options
  • Null handling (usenull=true) already implemented - Current timechart filters out nulls when determining top categories
  • Zero-filling for count aggregations already implemented - buildZeroFilledResult() method exists
  • aggregateWithTrimming function from visitAggregation is reused from stats. Since stats supports multiple aggregation functions, can reuse stats implementation to support multiple aggregations in timechart.
  • Timechart AST and parser infrastructure - Complete timechart implementation exists
Performance Considerations

See issue #4231 on timechart performance enhancement.

Documentation
  • docs/user/ppl/cmd/timechart.rst - Complete parameter documentation with examples

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing timechart implementation and docs/user/ppl/cmd/timechart.rst, then trace visitBin, BinUtils, BinHandlerFactory, aggregateWithTrimming, and buildZeroFilledResult. Reuse the existing stats support where specified and verify bins, usenull, cont, and multiple aggregations work together. Done means the documented syntax and requested time-series behaviors are supported consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.