apache / apache/parquet-java

filter2 API performance regression

Open
#1,583 17 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

The new filter API seems to be much slower (or perhaps I'm using it wrong \:)

Code using an UnboundRecordFilter:

```java
ColumnRecordFilter.column(column,
ColumnPredicates.applyFunctionToBinary(
input -> Binary.fromString(value).equals(input)));
```

vs. code using FilterPredicate:

```java
eq(binaryColumn(column), Binary.fromString(value));
```

The latter performs twice as slow on the same Parquet file (built using 1.6.0rc2).

Note: the reader is constructed using

```java
ParquetReader.builder(new ProtoReadSupport().withFilter(filter).build()
```

The new filter API based approach seems to create a whole lot more garbage (perhaps due to reconstructing all the rows?).

**Reporter**: [Viktor Szathmáry](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=phraktle) / @phraktle
#### Related issues:
- [FilteredRecordReader skips rows it shouldn't for schema with optional columns](https://github.com/apache/parquet-java/issues/1730) (is related to)

**Note**: *This issue was originally created as [PARQUET-98](https://issues.apache.org/jira/browse/PARQUET-98). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.