apache / apache/parquet-java

filter2 API performance regression

Open
#1,583 17 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

The new filter API seems to be much slower (or perhaps I'm using it wrong \:)

Code using an UnboundRecordFilter:

```java
ColumnRecordFilter.column(column,
ColumnPredicates.applyFunctionToBinary(
input -> Binary.fromString(value).equals(input)));
```

vs. code using FilterPredicate:

```java
eq(binaryColumn(column), Binary.fromString(value));
```

The latter performs twice as slow on the same Parquet file (built using 1.6.0rc2).

Note: the reader is constructed using

```java
ParquetReader.builder(new ProtoReadSupport().withFilter(filter).build()
```

The new filter API based approach seems to create a whole lot more garbage (perhaps due to reconstructing all the rows?).

**Reporter**: [Viktor Szathmáry](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=phraktle) / @phraktle
#### Related issues:
- [FilteredRecordReader skips rows it shouldn't for schema with optional columns](https://github.com/apache/parquet-java/issues/1730) (is related to)

**Note**: *This issue was originally created as [PARQUET-98](https://issues.apache.org/jira/browse/PARQUET-98). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the UnboundRecordFilter path using ColumnRecordFilter.column and ColumnPredicates.applyFunctionToBinary with the FilterPredicate path using eq and binaryColumn. Reproduce both cases on the same Parquet file with ProtoReadSupport and measure runtime and garbage creation; done means explaining and correcting the reported performance regression without changing filtering results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.