Write performance degradation between hudi 0.14 and 1.0.2
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**Environment:**
Spark 3.5
Java 11
Hudi 0.14 and 1.0.2
Storage S3
**What happened:**
We are noticing about 20-30% write performance degradation between Hudi 0.14 and 1.0.2
We have profiled the executors by writing the same data between executors running both versions, and here is the first of few suspects. We will open a separate bug report as we find more suspects.
Small file Handling:
In both versions, the HoodieMergeHelper.runMerge function, the following condition ends up being false, without any change to the writer schema during the test
```
// Check whether the writer schema is simply a projection of the file's one, ie
// - Its field-set is a proper subset (of the reader schema)
// - There's no schema evolution transformation necessary
boolean isPureProjection = schemaEvolutionTransformerOpt.isEmpty()
&& isStrictProjectionOf(readerSchema, writerSchema);
```
Due to this the record is always rewritten with function rewriteRecordWithNewSchema
```
executor = ExecutorFactory.create(writeConfig, recordIterator, new UpdateHandler(mergeHandle), record -> {
HoodieRecord newRecord;
if (schemaEvolutionTransformerOpt.isPresent()) {
newRecord = schemaEvolutionTransformerOpt.get().apply(record);
} else if (shouldRewriteInWriterSchema) {
newRecord = record.rewriteRecordWithNewSchema(recordSchema, writeConfig.getProps(), writerSchema);
} else {
newRecord = record;
}
```
In this function in hudi 1.x in HoodieAvroUtils.rewriteRecordWithNewSchema, the following check is added, which seems to be taking significant CPU time.
```
if (oldAvroSchema.equals(newSchema)) {
// there is no need to rewrite if the schema equals.
return oldRecord;
}
//
```
1.x flamegraph
0.14 flamegraph
hostspot in 1.0.2
**What you expected:**
No performance degradation
**Steps to reproduce:**
1. Write any dataset in 0.14 and 1.0.2 and observe the difference.
### Environment
**Hudi version: 0.14 and 1.0.2**
**Query engine: Spark** (Spark/Flink/Trino etc)
**Relevant configs: Same configurations as the one reported here: https://github.com/apache/hudi/issues/13995**
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported write comparison with Spark 3.5, Java 11, S3, and the same configurations and dataset on Hudi 0.14 and 1.0.2. Read HoodieMergeHelper.runMerge and HoodieAvroUtils.rewriteRecordWithNewSchema, then compare executor profiles to verify whether the schema equality check is the regression. Done means identifying the performance cause and validating the result with comparable measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100