apache / apache/hudi

[SUPPORT]hudi insert is too slow

Open
#10,131 6 comments 0 reactions 0 assignees View on GitHub
area:performance priority:high type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

spark sql bulk insert data is too slow , how to turn performance.
as https://hudi.apache.org/docs/performance
I do change many config, but is not well.

**To Reproduce**

Steps to reproduce the behavior:

1. spark-sql set hudi config
set hoodie.write.lock.zookeeper.lock_key=bi_ods_real.smart_datapoint_report_rw_clear_rt;
set spark.sql.hive.filesourcePartitionFileCacheSize=524288000;
set hoodie.metadata.table=false;
set hoodie.sql.insert.mode=non-strict;
set hoodie.sql.bulk.insert.enable=true;
set hoodie.populate.meta.fields=false;
set hoodie.parquet.compression.codec=snappy;

set hoodie.bloom.index.prune.by.ranges=false;
set hoodie.file.listing.parallelism=800;
set hoodie.cleaner.parallelism=800;
set hoodie.insert.shuffle.parallelism=800;
set hoodie.upsert.shuffle.parallelism=800;
set hoodie.delete.shuffle.parallelism=800;
set hoodie.memory.compaction.max.size= 4294967296;
set hoodie.memory.merge.max.size=107374182400;
2. sql
```
insert into bi_dw_real.dwd_smart_datapoint_report_rw_clear_rt
select
/*+ coalesce(${partitions}) */
md5(concat(coalesce(data_id,''),coalesce(dev_id,''),coalesce(gw_id,''),coalesce(product_id,''),coalesce(uid,''),coalesce(dp_code,''),coalesce(dp_id,''),if(dp_mode in ('ro','rw','wr'),dp_mode,'un'),coalesce(dp_name,''),coalesce(dp_time,''),coalesce(dp_type,''),coalesce(dp_value,''),coalesce(ct,''))) as id,
_hoodie_record_key as uuid,
data_id,dev_id,gw_id,product_id,uid,
dp_code,dp_id,if(dp_mode in ('ro','rw','wr'),dp_mode,'un') as dp_mode ,dp_name,dp_time,dp_type,dp_value,
ct as gmt_modified,
case
when length(ct)=10 then date_format(from_unixtime(ct),'yyyyMMddHH')
when length(ct)=13 then date_format(from_unixtime(ct/1000),'yyyyMMddHH')
else '1970010100' end as dt
from
hudi_table_changes('bi_ods_real.ods_log_smart_datapoint_report_batch_rt', 'latest_state', '${taskBeginTime}', '${next30minuteTime}')
lateral view dataPointExplode(split(value,'\001')[0]) dps as ct, data_id, dev_id, gw_id, product_id, uid, dp_code, dp_id, gmtModified, dp_mode, dp_name, dp_time, dp_type, dp_value
where _hoodie_commit_time >${taskBeginTime} and _hoodie_commit_time<=${next30minuteTime};
```
3. result table info
tblproperties (
type = 'mor',
primaryKey = 'id',
preCombineField = 'gmt_modified',
hoodie.combine.before.upsert='false',
hoodie.bucket.index.num.buckets=128,
hoodie.compact.inline='false',
hoodie.common.spillable.diskmap.type='ROCKS_DB',
hoodie.datasource.write.partitionpath.field='dt,dp_mode',
hoodie.compaction.payload.class='org.apache.hudi.common.model.PartialUpdateAvroPayload'
)
**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :0.14.0

* Spark version :3.2.1

* Hive version :3.2.1

* Hadoop version :3.2.2

* Storage (HDFS/S3/GCS..) :s3

* Running on Docker? (yes/no) :no

**Additional context**

![image](https://github.com/apache/hudi/assets/15028279/96c73e5a-d1f2-4db0-b583-37605ca754d0)

![image](https://github.com/apache/hudi/assets/15028279/1ce817b5-b372-4c09-a14c-f5ebf83f32fb)

how to change parallelism? I set spark-sql --conf spark.default.parallelism=800 is not work .

The follow config in sql file is not work as expect.

```
set hoodie.file.listing.parallelism=800;
set hoodie.cleaner.parallelism=800;
set hoodie.insert.shuffle.parallelism=800;
set hoodie.upsert.shuffle.parallelism=800;
set hoodie.delete.shuffle.parallelism=800;
```

The follow issues are not bulk insert .
#8189 #2620
Please take a look and give me some optimization suggestions.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Spark SQL bulk insert using Hudi 0.14.0, Spark 3.2.1, Hadoop 3.2.2, and S3 with the listed SQL and Hudi settings. Read the Hudi performance documentation and referenced issues #8189 and #2620, then determine why the parallelism settings have no visible effect and document actionable findings.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark, sql
Domain
data-engineering, databases, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.