[Feature] Log scene specialized index
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
I want to write the log data to the data lake for low-cost storage. The following indexes will be implemented:
TokenBloomFilter:Uses a Bloom filter to index individual tokens (words) extracted from a document or field. It allows fast existence checks, but has a small probability of false positives.
TokenBitmap:Indexes individual tokens using a bitmap where each bit represents the presence or absence of a token in a document set.
NgramBloomFilter:Applies a Bloom filter over N-grams (substrings of length N) extracted from text. Useful for fuzzy or substring matching with high performance.
NgramBitmap:Builds a bitmap index over all N-grams. Each bit corresponds to a document that contains the specific N-gram.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by locating the existing Java log-data and index implementations, then clarify the design and acceptance criteria for TokenBloomFilter, TokenBitmap, NgramBloomFilter, and NgramBitmap. Done should include the four specialized indexes writing log data to the data lake, with tests covering their behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100