duckdb / duckdb/duckdb-httpfs

Attempted to register an already registered secret type 's3'

Open
#31 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

### What happens?

Hello DuckDB team,

I would like to provide additional information regarding Issue duckdb/duckdb-httpfs#30.

Currently, when using a Presigned URL to read Parquet files from MinIO or other S3-compatible storages, I encounter the error Attempted to register an already registered secret type 's3' when the httpfs extension tries to handle simultaneous requests to the same URL.

Root cause of the issue:

- The issue seems related to concurrency, where multiple threads attempt to access the same Presigned URL concurrently. This causes the httpfs extension to attempt to register the S3 secret type more than once, resulting in the error.
- Since the Presigned URL already contains the authentication information, no additional S3 settings are required. However, when multiple requests are handled concurrently, the issue arises.

Other relevant notes:

- It may be necessary to improve the concurrency management and session handling to avoid conflicts.
- Consider implementing request queue management to prevent issues caused by simultaneous accesses.
I would appreciate it if this issue could be reviewed and addressed. Please let me know if you need any further information.

Thank you.

### To Reproduce

SQL
```
CREATE TABLE IF NOT EXISTS table AS SELECT * FROM read_parquet(presignedUrl)
```

JAVA
```
Connection conn = DriverManager.getConnection("jdbc:duckdb:");

String QUERY = "CREATE TABLE IF NOT EXISTS " + table + " AS SELECT * FROM read_parquet(?)";

try (PreparedStatement ps = conn.prepareStatement(QUERY)) {
ps.setString(1, url); // url is minio presigned url
return ps.execute();
}
```

### OS:

rocky linux 9.5

### DuckDB Version:

0.10.0

### DuckDB Client:

java 21

### Hardware:

_No response_

### Full Name:

jinseo kim

### Affiliation:

developer

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have not tested with any build

### Did you include all relevant data sets for reproducing the issue?

No - Other reason (please specify in the issue body)

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

- [ ] Yes, I have

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the httpfs extension path used by read_parquet with a presigned S3-compatible URL, and reproduce the error with simultaneous SQL or Java JDBC requests. The fix is done when concurrent requests no longer attempt to register the same 's3' secret type and the supplied reproduction succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java, sql
Domain
cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.