Intermittent `KeyError` when writing to S3
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
Still experiencing intermittent Access Denied errors see #218.
However today a new one has popped up when writing to_parquet. In this case:
df.to_parquet(
'path/to/20190719.parquet',
engine='fastparquet',
compression='snappy',
times='int96', # spark supported timestamps
storage_options={'s3_additional_kwargs': server_side_encryption_args},
)
occasionally (with a similar freq to access denied so far) is giving:
'path/to/20190719.parquet'
Traceback (most recent call last):
File "<ipython-input-50-e89c9bf2fe97>", line 16, in <module>
storage_options={'s3_additional_kwargs': sse_args},
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/dask/dataframe/core.py", line 3618, in to_parquet
return to_parquet(self, path, *args, **kwargs)
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/dask/dataframe/io/parquet.py", line 1496, in to_parquet
out.compute()
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/dask/base.py", line 175, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/dask/base.py", line 446, in compute
results = schedule(dsk, keys, **kwargs)
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/client.py", line 2527, in get
results = self.gather(packed, asynchronous=asynchronous, direct=direct)
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/client.py", line 1823, in gather
asynchronous=asynchronous,
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/client.py", line 763, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/utils.py", line 332, in sync
six.reraise(*error[0])
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/utils.py", line 317, in f
result[0] = yield future
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/client.py", line 1680, in _gather
six.reraise(type(exception), exception, traceback)
File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/dask/dataframe/io/parquet.py", line 617, in _read_parquet_row_group
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/fastparquet/core.py", line 302, in read_row_group_file
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/fsspec/spec.py", line 689, in open
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 303, in _open
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 920, in __init__
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/fsspec/spec.py", line 884, in __init__
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 479, in info
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/fsspec/spec.py", line 495, in info
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 497, in ls
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 430, in _ls
File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 339, in _lsdir
KeyError: 'path/to/20190719.parquet'
environment is:
- dask 2.1.0
- distributed 2.1.0
- s3fs 3.3.0
- fsspec 0.4.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the intermittent failure with the listed dask, distributed, s3fs, and fsspec versions using the shown to_parquet call. Start in s3fs/core.py at _lsdir and follow the fsspec info/open path from the traceback; done means the KeyError no longer occurs for this S3 write/read flow and the behavior is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100