DuckDB 2.0 httpfs multipart upload - Cloudflare R2 InvalidPart
- Dominant language
- C++
- Stars
- 60
- Forks
- 100
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 25
Description
Copying a bug report [from Discord](https://discord.com/channels/909674491309850675/1540958679141191750/1541508882382323864):
**Summary**
Large `COPY … TO 's3://…' (FORMAT PARQUET)` against **Cloudflare R2** fails at `CompleteMultipartUpload` with:
```text
InvalidPart: All non-trailing parts must have the same length
```
R2 requires equal size for all non-trailing multipart parts. DuckDB’s httpfs uploader appears to emit uneven part sizes.
**Repro (Docker, linux/amd64)**
1. Official preview CLI: `https://artifacts.duckdb.org/latest/duckdb-cli-linux-amd64.tar.gz`
→ `v2.0.0-alpha38837` (`42b8522dc4`)
2. Also seen on Grain’s fork of `v2.0.0-alpha38615` (same failure → not fork-specific)
3. `INSTALL httpfs; LOAD httpfs;` + S3 secret (`ENDPOINT` = `*.r2.cloudflarestorage.com`, `URL_STYLE 'path'`, `REGION 'auto'` or `'us-east-1'`)
4. `COPY` ~2–3M rows to `s3://bucket/…/file.parquet`
**Not the cause**
- `REGION 'auto'` vs `'us-east-1'` — both fail
- `httpfs_connection_caching` on/off — both fail
- DuckDB 1.5.5 against the same R2 bucket — **works**
**Workaround**
Force single-part upload, e.g. `SET s3_uploader_max_parts_per_file=1;` (or keep object under one part). Tiny files / single-PUT succeed.
**Ask**
Can httpfs use fixed part sizes for S3-compatible stores (R2), or document a supported setting so multipart writes to R2 don’t violate equal-part rules?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure with the Docker Linux/amd64 CLI, INSTALL/LOAD httpfs, and a large COPY ... TO s3://... (FORMAT PARQUET) against Cloudflare R2. Trace the httpfs multipart upload entry point and compare the preview behavior with DuckDB 1.5.5, which succeeds. Done means multipart uploads satisfy R2’s equal non-trailing-part requirement, or a supported setting for R2 is documented and verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100