apache / apache/lucene

count aggregation optimization inside one segment in log scenario [LUCENE-10425]

Open
#11,461 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In log scenario, we usually want to know the doc count of documents between every time intervals. One possible optimized method is to sort the docuemt in ascend order according to `@timestamp` field in one segment. then we can use    this pr  to find out the min/max docId in on time interval.

If there is no other filter query, the doc count of one time interval is (max docId- min docId +1)

if there is only one another term filter query, we can use this pr [https://github.com/apache/lucene/pull/688 ](https://github.com/apache/lucene/pull/688)to get the diff value of index, when we call advance(minId) and advance(maxId), the diff value is also the doc count of one time interval

---
Migrated from [LUCENE-10425](https://issues.apache.org/jira/browse/LUCENE-10425) by jianping weng (@wjp719), updated Mar 29 2022

Contributor guide

Open the contributing guide

Research direction

Start by reading the log scenario context and the referenced Lucene pull requests #687 and #688, which describe doc ID range and iterator advancement behavior. The issue does not name files or tests and does not define a concrete acceptance criterion, so the implementation scope and definition of done need clarification.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.