apache / apache/paimon

[Bug] Paimon s3 configuration conflicts with Flink s3 configuration

Open
#1,898 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

paimon-flink-1.17-0.5-20230814.001854-121.jar

### Compute Engine

Flink Version:1.17.1

### Minimal reproduce step

I use paimon with s3 storage and configure s3 access through Flink FlleSystem(flink-s3-fs-hadoop-1.17.1.jar).
I want to use different s3 storage to configure paimon directory and savpoint and checkpoint of flink

My s3 configuration in flink is
s3.access-key=access-key1
s3.secret-key=serect-key1
s3.endpoint=http://endpoint1:80

My paimon cataglog ddl:
```sql
CREATE CATALOG my_catalog WITH (
'type' = 'paimon',
'warehouse' = 's3a://mybucket/',
's3.endpoint' = 'http://endpoint2:80',
's3.access-key' = 'serect-key2',
's3.secret-key' = 'access-key2'
);
```
However, It' seem like some confliction happen.

### What doesn't meet your expectations?

![6a2a0aace8be5e4bc2681dc70e97a2f](https://github.com/apache/incubator-paimon/assets/21351694/c8657d7a-e6dd-43d8-9fc9-79f05add4205)
![60e73703d26c288381ee51ea7410239](https://github.com/apache/incubator-paimon/assets/21351694/4e7cf759-9837-40bf-ae21-a5d7a36b8f0b)

It seems that when paimon accesses s3, it uses the Flink configuration instead of the configuration in the catalog.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Paimon catalog DDL and the Flink S3 filesystem configuration described in the reproduction, then trace where catalog options and Flink settings meet during S3 access. Reproduce the setup with separate endpoints and credentials; done means Paimon uses the catalog configuration without conflicting with Flink's checkpoint and savepoint configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.