apache / apache/paimon

[Feature] Log scene specialized index

Open
#5,558 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

I want to write the log data to the data lake for low-cost storage. The following indexes will be implemented:

TokenBloomFilter:Uses a Bloom filter to index individual tokens (words) extracted from a document or field. It allows fast existence checks, but has a small probability of false positives.

TokenBitmap:Indexes individual tokens using a bitmap where each bit represents the presence or absence of a token in a document set.

NgramBloomFilter:Applies a Bloom filter over N-grams (substrings of length N) extracted from text. Useful for fuzzy or substring matching with high performance.

NgramBitmap:Builds a bitmap index over all N-grams. Each bit corresponds to a document that contains the specific N-gram.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by locating the existing Java log-data and index implementations, then clarify the design and acceptance criteria for TokenBloomFilter, TokenBitmap, NgramBloomFilter, and NgramBitmap. Done should include the four specialized indexes writing log data to the data lake, with tests covering their behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.