drivendataorg / drivendataorg/cloudpathlib

S3 Paths change local cache dir after pickling

Open
#504 4 comments 0 reactions 0 assignees View on GitHub
caching
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

Hey,

thanks for the awesome library!

# Bug
I noticed that S3Paths change their local cache dir of the attached S3Client when pickled.
The S3Client itself appears not to be pickleable due to boto sessions not being pickleable, however, the S3Path even with explicitly specified client is pickleable without throwing errors or warnings.
This is a major problem in multiprocessing for me where pickling is necessary. Here files are then downloaded again instead of using the cached version.

I saw https://github.com/drivendataorg/cloudpathlib/issues/450 but encounter the error both for version 0.19 and 0.20 (python 3.10.16)

## Steps to reproduce
```python
import cloudpathlib as cl

# get s3 client from somewhere
def test_s3_pickled_client(s3_client: cl.S3Client) -> None:
path = cl.S3Path("s3://dummy", client=s3_client)
path_re = pickle.loads(pickle.dumps(path))

assert path_re.client._local_cache_dir == path.client._local_cache_dir # fails
```

## Environment
```bash
boto3 1.36.5
botocore 1.36.5
cloudpathlib 0.20.0

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.