Add `multipart_chunksize` / `multipart_threshold` to the `s3` remote config schema
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Problem
dvc-s3 already maps multipart_chunksize and multipart_threshold into boto3's TransferConfig via _TRANSFER_CONFIG_ALIASES (see dvc_s3/__init__.py lines 59–60, 117), but dvc/config_schema.py's s3 block (lines ~165–193) doesn't list either key. Result: any attempt to set them is rejected at config-load time, before the plugin ever sees them.
$ dvc remote modify --local codap-dvc multipart_chunksize 64MiB
ERROR: configuration error - config file error: extra keys not allowed @ data['remote']['PROFILE']['multipart_chunksize']
Reproduced on DVC 3.66.1 and 3.67.1.
Why this matters
Scaleway Object Storage caps multipart uploads at 1000 parts (vs AWS S3's 10,000). With boto3's default 8 MiB part size, the maximum object size is 8 MiB × 1000 = 8 GiB. Any file larger than that fails with:
ERROR: failed to transfer '<hash>' - [Errno 22] Part number must be an integer between 1 and 1000, inclusive: An error occurred (InvalidArgument) when calling the UploadPart operation
The intended fix — multipart_chunksize 64MiB, lifting the ceiling to 64 GiB — is correctly handled by dvc-s3 once the value reaches the plugin, but the config schema blocks it from ever getting there.
This is not Scaleway-specific in any architectural sense; the same gap blocks any S3-compatible backend with stricter multipart limits, and blocks legitimate tuning on AWS itself for very large objects.
Proposed fix
Add the two keys to the s3 block in dvc/config_schema.py:
"s3": {
# ...existing keys...
"read_timeout": Coerce(int),
"connect_timeout": Coerce(int),
"multipart_threshold": str, # human-readable size, e.g. "64MiB"
"multipart_chunksize": str, # human-readable size, e.g. "64MiB"
Optional("verify", default=False): Bool,
**REMOTE_COMMON,
},
dvc-s3's _split_s3_config already calls human_readable_to_bytes on these two keys, so accepting str is correct. Optionally, max_queue_size and max_concurrent_requests (also in the alias map but not the schema) could be added in the same change — same gap, same rationale.
Workaround
For users blocked today, content-addressed objects can be uploaded directly via rclone (or any S3 client) with a sufficient chunk_size, written to the same <bucket>/files/md5/<2chars>/<rest> path DVC uses. dvc push becomes a HEAD-check no-op for those objects; dvc pull works transparently from any other machine since the path layout is the contract, not the upload mechanism. This isn't ideal — every operator with a >8 GiB object hits it — but it unblocks the immediate failure.
Environment
- DVC: 3.66.1 (also reproduced on 3.67.1)
- dvc-s3: 3.3.0
- aiobotocore: 3.6.0, botocore: 1.43.0
- Python 3.13.2, Linux
- Remote: Scaleway Object Storage (S3-compatible,
s3.fr-par.scw.cloud)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in dvc/config_schema.py, inspecting the s3 block and compare its accepted keys with the aliases in dvc_s3/init.py. Add schema coverage for the two multipart settings, then run the relevant configuration validation tests; done means values such as 64MiB load without an extra-key error and reach the plugin.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100