DuckLake-opened S3 files skip bucket-region resolution since 1.5.3; 1.5.5 masks the 301 with misleading errors
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
### What happens
With an S3 secret that has no `REGION` and a bucket outside us-east-1, scanning a DuckLake table fails, while `read_parquet` on the same objects with the same secret works on every version:
| duckdb | DuckLake table scan |
|---|---|
| 1.5.2 | works |
| 1.5.3 / 1.5.4 | `HTTP Error: Request returned HTTP 301` |
| 1.5.5 | `HTTP Error: Server sent back more data than expected, 'SET force_download=true' might help in this case` |
| 1.5.5 + `force_download` | `Error: don't know what type: ` on any query that reads a file; metadata-answered `count(*)` still works |
`CALL enable_logging('HTTP')` shows the data-file requests signed for `.s3.us-east-1.amazonaws.com` although the bucket is in another region, so S3 answers with a 301 XML body. On 1.5.5 that body is larger than the requested parquet-footer range ("more data than expected"), and with `force_download` it is saved as the "parquet file", hence the nonsense downstream error. `threads=1` changes nothing. The AWS SDK config in use does set `region`; it is ignored on this path, and only an explicit `REGION` on the secret changes the outcome.
### Workaround
`CREATE SECRET s (TYPE S3, PROVIDER CREDENTIAL_CHAIN, REGION '')` fixes all versions.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the DuckLake table-scan path that opens S3 data files and compare it with read_parquet using the same secret. Use CALL enable_logging('HTTP') to verify which region signs the requests; done means scans resolve the bucket region without an explicit secret REGION and no longer produce the 301 or misleading file errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100