drivendataorg / drivendataorg/cloudpathlib

For users missing certain S3 permissions, files may be identified as directories

Open
#310 3 comments 1 reaction 0 assignees View on GitHub
bug S3
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

Depending on the machine in which the upload happens, downloading works fine. However, when a different machine tries to download the S3 remote path, it downloads files as directory.

```python
from pathlib import Path

from cloudpathlib import CloudPath

CWD = Path.cwd()
TMP = Path(tempfile.gettempdir())

remote_path = CloudPath(f's3://test/models/')

local_path = CWD / 'models/'
remote_path.upload_from(local_path)

for rp in remote_path.iterdir():
lp = local_path / rp.relative_to(remote_path)
assert rp.is_file() == lp.is_file(), f'The {rp} remote path must be a file!'
assert rp.is_dir() == lp.is_dir(), f'The {rp} remote path must be a directory!'

local_path = TMP / 'models'
remote_path.download_to(local_path)

for rp in remote_path.iterdir():
lp = local_path / rp.relative_to(remote_path)
assert rp.is_file() == lp.is_file(), f'The {lp} local path must be a file!'
assert rp.is_dir() == lp.is_dir(), f'The {lp} local path must be a directory!'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.