apache / apache/paimon

[Feature] When writing using Spark, support for write parallelism

Open
#4,742 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

## Background
There exists a table that satisfies the following conditions:

1. The number of buckets is large

2. The amount of data is huge

3. changelog=lookup

I use Spark sql (batch) to read the hive table and write data into it. If the amount of data I read is not large, only less parallelism p1 will be generated during the split phase of spark scan.

The final write to paimon will also use p1 parallelism:

image

This will cause problems. The degree of parallelism is too small. When writing, building lookupcache will be blocked by io, which will cause the task to run for a long time even if only a little data is written.

### Solution

It is necessary to be able to set the parallelism of writing, and at the same time to automatically infer the number of buckets. The default setting of writing parallelism is the number of buckets.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Spark SQL batch read/write path described in the issue and trace how scan parallelism reaches the Paimon write. Determine where write parallelism and bucket-count inference should be configured, then verify that the default uses the bucket count and that writes avoid the reported lookup-cache I/O bottleneck.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.