treeverse / treeverse/dvc

Support virtual-hosted–style only S3-compatible remote?

Open
#10,256 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature request fs: s3
Dominant language
Python
Stars
15.9k
Forks
1.3k
No merged PRs in 30d

Description

Does dvc support virtual-hosted–style S3-compatible remote?

From doc, dvc seem dvc only support path-style S3 remote.

When I try to use dvc with Tinder Object Storage (TOS) which only support virtual-hosted–style S3, it report error.

  • BytePlus | Compatibility with Amazon S3 link
  • Virtual hosting of buckets - Amazon Simple Storage Service link

Error info:

$ cat .dvc/config
[core]
    remote = tos
['remote "tos"']
    url = s3://xxx/xxx (edited)
    endpointurl = https://tos-s3-xxx.xxxx.com （edited）
$ dvc push 
Collecting                                                                                                                                                         |0.00 [00:00,    ?entry/s]
Pushing
ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden                                                                                

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
$ dvc push --verbose
2024-01-25 17:21:30,113 DEBUG: v3.42.0 (brew), CPython 3.12.1 on macOS-14.2.1-arm64-arm-64bit
2024-01-25 17:21:30,113 DEBUG: command: /opt/homebrew/bin/dvc push --verbose
Collecting                                                                                                                                                         |0.00 [00:00,    ?entry/s]
2024-01-25 17:21:30,346 DEBUG: Preparing to transfer data from 'x'
2024-01-25 17:21:30,346 DEBUG: Preparing to collect status from 'x'
2024-01-25 17:21:30,346 DEBUG: Collecting status from 'x'
2024-01-25 17:21:30,347 DEBUG: Querying 1 oids via object_exists
Pushing                                                                                                                                                                                      
2024-01-25 17:21:30,689 ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden                                                        
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 113, in _error_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/aiobotocore/client.py", line 408, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
          ^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/cli/command.py", line 27, in do_run
    return self.run()
           ^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/commands/data_sync.py", line 64, in run
    processed_files_count = self.repo.push(
                            ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/repo/__init__.py", line 65, in wrapper
    return f(repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc/repo/push.py", line 144, in push
    push_transferred, push_failed = ipush(
                                    ^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_data/index/push.py", line 75, in push
    result = transfer(
             ^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_data/hashfile/transfer.py", line 203, in transfer
    status = compare_status(
             ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_data/hashfile/status.py", line 178, in compare_status
    dest_exists, dest_missing = status(
                                ^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_data/hashfile/status.py", line 150, in status
    exists.update(odb.oids_exist(hashes, jobs=jobs, progress=pbar.callback))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/db.py", line 422, in oids_exist
    return list(wrap_iter(remote_oids, callback))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/db.py", line 36, in wrap_iter
    for index, item in enumerate(iterable, start=1):
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/db.py", line 370, in list_oids_exists
    in_remote = self.fs.exists(paths, batch_size=jobs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/fs/base.py", line 472, in exists
    return fut.result()
           ^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/dvc_objects/executors.py", line 135, in batch_coros
    result = fut.result()
             ^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 1035, in _exists
    await self._info(path, bucket, key, version_id=version_id)
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 1302, in _info
    out = await self._call_s3(
          ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 348, in _call_s3
    return await _error_wrapper(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.42.0/libexec/lib/python3.12/site-packages/s3fs/core.py", line 140, in _error_wrapper
    raise err
PermissionError: Forbidden

2024-01-25 17:21:30,721 DEBUG: Version info for developers:
DVC version: 3.42.0 (brew)
--------------------------
Platform: Python 3.12.1 on macOS-14.2.1-arm64-arm-64bit
Subprojects:
	dvc_data = 3.8.0
	dvc_objects = 3.0.6
	dvc_render = 1.0.1
	dvc_task = 0.3.0
	scmrepo = 2.0.4
Supports:
	azure (adlfs = 2023.12.0, knack = 0.11.0, azure-identity = 1.15.0),
	gdrive (pydrive2 = 1.19.0),
	gs (gcsfs = 2023.12.2.post1),
	http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	oss (ossfs = 2023.12.0),
	s3 (s3fs = 2023.12.2, boto3 = 1.34.22),
	ssh (sshfs = 2023.10.0),
	webdav (webdav4 = 0.9.8),
	webdavs (webdav4 = 0.9.8),
	webhdfs (fsspec = 2023.12.2)
Config:
	Global: /Users/bytedance/Library/Application Support/dvc
	System: /opt/homebrew/share/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s3s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk3s3s1
Repo: dvc, git
Repo.site_cache_dir: /opt/homebrew/var/cache/dvc/repo/b8147adaf473f039b47ac961430c05d4

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2024-01-25 17:21:30,725 DEBUG: Analytics is enabled.
2024-01-25 17:21:30,755 DEBUG: Trying to spawn ['daemon', 'analytics', '/var/folders/x3/m9b0vhpn28d_557yb666lf4w0000gn/T/tmpay2t_qwd', '-v']
2024-01-25 17:21:30,762 DEBUG: Spawned ['daemon', 'analytics', '/var/folders/x3/m9b0vhpn28d_557yb666lf4w0000gn/T/tmpay2t_qwd', '-v'] with pid 98616

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the S3 remote path exercised by dvc push, tracing from dvc/repo/push.py through the s3fs and boto3 integration shown in the report. Reproduce the 403 against a virtual-hosted-only endpoint, then verify that push works and update the linked S3-compatible-server documentation with the supported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.