apache / apache/parquet-java

Improve performance when filters in RowGroupFilter can match exactly

Offen
#1,655 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Component: Parquet Priority: Major Type: enhancement
Vorherrschende Sprache
Java
Sterne
3.1k
Forks
1.6k
Ø Merge
3 T. 12 Std.
Gemergte PRs (30 T.)
33

Beschreibung

If we can accurately judge by the minMax status, we don’t need to load the dictionary from filesystem and compare one by one anymore.

Similarly , Bloomfilter needs to load from filesystem, it may costs time and memory. If we can exactly determine the existence/nonexistence of the value from minMax or dictionary filters , then we can avoid using Bloomfilter to Improve performance.

For example,
1. read data greater than `x1` in the block, if minMax in status is all greater than `{}x1{`}, then we don't need to read dictionary and compare one by one.
1. If we already have page dictionaries and have compared one by one, we don't need to read BloomFilter and compare.

**Reporter**: [Mars](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=miracle) / @yabola
**Assignee**: [Mars](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=miracle) / @yabola

**Note**: *This issue was originally created as [PARQUET-2237](https://issues.apache.org/jira/browse/PARQUET-2237). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start by locating RowGroupFilter and tracing how minMax, page dictionaries, and Bloomfilter are evaluated. Confirm how exact minMax or dictionary results can skip later filesystem reads, then verify the behavior and performance with the relevant existing checks; the issue names no files or tests, so those must be identified first.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
data-engineering
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.