apache / apache/hudi

[SUPPORT] Incremental query performance

Open
#7,254 7 comments 0 reactions 0 assignees View on GitHub
area:performance priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Hi all,
I have a question about the performance of incremental queries. I'm comparing the performance between running incremental queries and simply doing a filter on the `_hoodie_commit_time` column. From initial investigations, it looks like for deltas with a small number of commits, incremental queries are more performant but as we increase the number of commits, the incremental queries actually take longer to complete compared to a column filter. Does anyone know why I'd be seeing this behaviour and whether it's expected? My expectation would have been that incremental queries will be more performant than filters in all scenarios as it would be scanning less data.

here are some results of the performance of the two:
```
1 commit 10 commits 20 commits
Incremental 2'05" 3'12" 4'28"
Filter 2'55" 2'51" 3'03
```
attaching code snippets for reference:
```
incremental queries:
beginTime = '20220928105015966'

incremental_read_options = {
'hoodie.datasource.query.type': 'incremental',
'hoodie.datasource.read.begin.instanttime': beginTime,
}

df = spark.read.format("hudi"). \
options(**incremental_read_options). \
load("s3://bucketpath/")

df.groupby("column_name").count().collect()

filter query:
df = spark.read.format("hudi"). \
load("s3://bucketpath/")

df.where("_hoodie_commit_time >= '20220928105015966'").groupby("column_name").count().collect()
```
**To Reproduce**

Steps to reproduce the behavior:

1. Create an EMR Serverless Job on AWS
2. Run the code specified in the section above

**Expected behavior**

Incremental queries are more performant than filter queries

**Environment Description**

* Hudi version : 0.11.1-amzn-0

* Spark version : 3.3.0

* Hive version : 3.1.3

* Hadoop version : 3.2.1

* Storage (HDFS/S3/GCS..) : N/A

* Running on Docker? (yes/no) : no

**Additional context**

N/A

**Stacktrace**

N/A

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the supplied Spark snippets using hoodie.datasource.query.type=incremental and the _hoodie_commit_time filter, then reproduce the comparison in an EMR Serverless job with the stated Hudi, Spark, Hive, and Hadoop versions. Compare execution plans and timings across 1, 10, and 20 commits; done means explaining the performance difference or documenting a confirmed expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, spark
Domain
cloud, data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.