treeverse / treeverse/dvc

Add `multipart_chunksize` / `multipart_threshold` to the `s3` remote config schema

Open Beginner friendly
#11,034 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Problem

dvc-s3 already maps multipart_chunksize and multipart_threshold into boto3's TransferConfig via _TRANSFER_CONFIG_ALIASES (see dvc_s3/__init__.py lines 59–60, 117), but dvc/config_schema.py's s3 block (lines ~165–193) doesn't list either key. Result: any attempt to set them is rejected at config-load time, before the plugin ever sees them.

$ dvc remote modify --local codap-dvc multipart_chunksize 64MiB
ERROR: configuration error - config file error: extra keys not allowed @ data['remote']['PROFILE']['multipart_chunksize']

Reproduced on DVC 3.66.1 and 3.67.1.

Why this matters

Scaleway Object Storage caps multipart uploads at 1000 parts (vs AWS S3's 10,000). With boto3's default 8 MiB part size, the maximum object size is 8 MiB × 1000 = 8 GiB. Any file larger than that fails with:

ERROR: failed to transfer '<hash>' - [Errno 22] Part number must be an integer between 1 and 1000, inclusive: An error occurred (InvalidArgument) when calling the UploadPart operation

The intended fix — multipart_chunksize 64MiB, lifting the ceiling to 64 GiB — is correctly handled by dvc-s3 once the value reaches the plugin, but the config schema blocks it from ever getting there.

This is not Scaleway-specific in any architectural sense; the same gap blocks any S3-compatible backend with stricter multipart limits, and blocks legitimate tuning on AWS itself for very large objects.

Proposed fix

Add the two keys to the s3 block in dvc/config_schema.py:

"s3": {
    # ...existing keys...
    "read_timeout": Coerce(int),
    "connect_timeout": Coerce(int),
    "multipart_threshold": str,    # human-readable size, e.g. "64MiB"
    "multipart_chunksize": str,    # human-readable size, e.g. "64MiB"
    Optional("verify", default=False): Bool,
    **REMOTE_COMMON,
},

dvc-s3's _split_s3_config already calls human_readable_to_bytes on these two keys, so accepting str is correct. Optionally, max_queue_size and max_concurrent_requests (also in the alias map but not the schema) could be added in the same change — same gap, same rationale.

Workaround

For users blocked today, content-addressed objects can be uploaded directly via rclone (or any S3 client) with a sufficient chunk_size, written to the same <bucket>/files/md5/<2chars>/<rest> path DVC uses. dvc push becomes a HEAD-check no-op for those objects; dvc pull works transparently from any other machine since the path layout is the contract, not the upload mechanism. This isn't ideal — every operator with a >8 GiB object hits it — but it unblocks the immediate failure.

Environment

  • DVC: 3.66.1 (also reproduced on 3.67.1)
  • dvc-s3: 3.3.0
  • aiobotocore: 3.6.0, botocore: 1.43.0
  • Python 3.13.2, Linux
  • Remote: Scaleway Object Storage (S3-compatible, s3.fr-par.scw.cloud)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dvc/config_schema.py, inspecting the s3 block and compare its accepted keys with the aliases in dvc_s3/init.py. Add schema coverage for the two multipart settings, then run the relevant configuration validation tests; done means values such as 64MiB load without an extra-key error and reach the plugin.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.