fsspec / fsspec/s3fs

Intermittent `KeyError` when writing to S3

Open
#233 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

Still experiencing intermittent Access Denied errors see #218.

However today a new one has popped up when writing to_parquet. In this case:

df.to_parquet(
    'path/to/20190719.parquet',
    engine='fastparquet',
    compression='snappy',
    times='int96',  # spark supported timestamps  
    storage_options={'s3_additional_kwargs': server_side_encryption_args},
)

occasionally (with a similar freq to access denied so far) is giving:

'path/to/20190719.parquet'
Traceback (most recent call last):
  File "<ipython-input-50-e89c9bf2fe97>", line 16, in <module>
    storage_options={'s3_additional_kwargs': sse_args},
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/dask/dataframe/core.py", line 3618, in to_parquet
    return to_parquet(self, path, *args, **kwargs)
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/dask/dataframe/io/parquet.py", line 1496, in to_parquet
    out.compute()
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/dask/base.py", line 175, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/dask/base.py", line 446, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/client.py", line 2527, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/client.py", line 1823, in gather
    asynchronous=asynchronous,
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/client.py", line 763, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/utils.py", line 332, in sync
    six.reraise(*error[0])
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/utils.py", line 317, in f
    result[0] = yield future
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/distributed/client.py", line 1680, in _gather
    six.reraise(type(exception), exception, traceback)
  File "/mnt/miniconda/envs/jestr-etl/lib/python3.7/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/dask/dataframe/io/parquet.py", line 617, in _read_parquet_row_group
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/fastparquet/core.py", line 302, in read_row_group_file
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/fsspec/spec.py", line 689, in open
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 303, in _open
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 920, in __init__
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/fsspec/spec.py", line 884, in __init__
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 479, in info
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/fsspec/spec.py", line 495, in info
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 497, in ls
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 430, in _ls
  File "/mnt1/yarn/usercache/hadoop/appcache/application_1566335097751_3990/container_1566335097751_3990_01_000015/environment/lib/python3.7/site-packages/s3fs/core.py", line 339, in _lsdir
KeyError: 'path/to/20190719.parquet'

environment is:

  • dask 2.1.0
  • distributed 2.1.0
  • s3fs 3.3.0
  • fsspec 0.4.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the intermittent failure with the listed dask, distributed, s3fs, and fsspec versions using the shown to_parquet call. Start in s3fs/core.py at _lsdir and follow the fsspec info/open path from the traceback; done means the KeyError no longer occurs for this S3 write/read flow and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.