duckdb / duckdb/duckdb-httpfs

DuckDB 2.0 httpfs multipart upload - Cloudflare R2 InvalidPart

Open
#405 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
60
Forks
100
Avg merge
1h 50m
Merged PRs (30d)
25

Description

Copying a bug report [from Discord](https://discord.com/channels/909674491309850675/1540958679141191750/1541508882382323864):

**Summary**
Large `COPY … TO 's3://…' (FORMAT PARQUET)` against **Cloudflare R2** fails at `CompleteMultipartUpload` with:

```text
InvalidPart: All non-trailing parts must have the same length
```

R2 requires equal size for all non-trailing multipart parts. DuckDB’s httpfs uploader appears to emit uneven part sizes.

**Repro (Docker, linux/amd64)**
1. Official preview CLI: `https://artifacts.duckdb.org/latest/duckdb-cli-linux-amd64.tar.gz`
→ `v2.0.0-alpha38837` (`42b8522dc4`)
2. Also seen on Grain’s fork of `v2.0.0-alpha38615` (same failure → not fork-specific)
3. `INSTALL httpfs; LOAD httpfs;` + S3 secret (`ENDPOINT` = `*.r2.cloudflarestorage.com`, `URL_STYLE 'path'`, `REGION 'auto'` or `'us-east-1'`)
4. `COPY` ~2–3M rows to `s3://bucket/…/file.parquet`

**Not the cause**
- `REGION 'auto'` vs `'us-east-1'` — both fail
- `httpfs_connection_caching` on/off — both fail
- DuckDB 1.5.5 against the same R2 bucket — **works**

**Workaround**
Force single-part upload, e.g. `SET s3_uploader_max_parts_per_file=1;` (or keep object under one part). Tiny files / single-PUT succeed.

**Ask**
Can httpfs use fixed part sizes for S3-compatible stores (R2), or document a supported setting so multipart writes to R2 don’t violate equal-part rules?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure with the Docker Linux/amd64 CLI, INSTALL/LOAD httpfs, and a large COPY ... TO s3://... (FORMAT PARQUET) against Cloudflare R2. Trace the httpfs multipart upload entry point and compare the preview behavior with DuckDB 1.5.5, which succeeds. Done means multipart uploads satisfy R2’s equal non-trailing-part requirement, or a supported setting for R2 is documented and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.