drivendataorg / drivendataorg/cloudpathlib
S3 Paths change local cache dir after pickling
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
Hey,
thanks for the awesome library!
# Bug
I noticed that S3Paths change their local cache dir of the attached S3Client when pickled.
The S3Client itself appears not to be pickleable due to boto sessions not being pickleable, however, the S3Path even with explicitly specified client is pickleable without throwing errors or warnings.
This is a major problem in multiprocessing for me where pickling is necessary. Here files are then downloaded again instead of using the cached version.
I saw https://github.com/drivendataorg/cloudpathlib/issues/450 but encounter the error both for version 0.19 and 0.20 (python 3.10.16)
## Steps to reproduce
```python
import cloudpathlib as cl
# get s3 client from somewhere
def test_s3_pickled_client(s3_client: cl.S3Client) -> None:
path = cl.S3Path("s3://dummy", client=s3_client)
path_re = pickle.loads(pickle.dumps(path))
assert path_re.client._local_cache_dir == path.client._local_cache_dir # fails
```
## Environment
```bash
boto3 1.36.5
botocore 1.36.5
cloudpathlib 0.20.0
```
Contributor guide
Assessment
This issue has not been assessed yet.