apache / apache/fluss

The local directory of the container should be used as the temporary directory of fluss client when the Flink job is deployed to YARN

Open
#2,235 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.8.0 (latest release)

### Please describe the bug 🐞

I deploy a flink streaming job to YARN. This job reads some fluss tables. It's taskmanager (YARN container) use /tmp/fluss as the temporary directory. I think that the local directory of the container should typically be used as the temporary directory of fluss read client.

### Solution

Use one directory in the option 'io.tmp.dirs' of flink configuration to set the option ‘client.scanner.io.tmpdir’ of fluss configuration after the taskmanager startup

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at Flink TaskManager startup and trace how the Flink configuration option `io.tmp.dirs` is made available to the Fluss client. Verify that `client.scanner.io.tmpdir` uses a directory from that setting instead of `/tmp/fluss` in YARN containers, and confirm the behavior with the relevant client and deployment tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.