opensearch-project / opensearch-project/sql

[FEATURE] Support "consecutive=true" argument in `dedup` command with Calcite

Open
#3,438 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

calcite enhancement
Dominant language
Java
Stars
176
Forks
229
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Is your feature request related to a problem?
Currently, we implements dedup by row number window function. The argument "consecutive=true" required the input data forced with certain ordering.
For example, only following ppl have business meaning.
source=t | sort timestamp | dedup 2 col1 consecutive=true
or
source=t | dedup 2 col1 consecutive=true sort by timestamp

query without sorting, such as
source=t | dedup 2 col1 consecutive=true
should throw exception "dedupe data with consecutive has no ordering"

What solution would you like?
A clear and concise description of what you want to happen.

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Calcite-backed planning path for the dedup command and reviewing how its row-number window and input ordering are handled. Find or add relevant dedup and ordering tests; done means consecutive=true works with an explicit ordering and an unordered query raises the specified exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.