fsspec / fsspec/s3fs

s3fs 2024.3.0 fails reading glob patterns through pandas

Open
#862 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

Intention is to read all CSV files under an s3 directory using Pandas via s3fs.
We specify path as s3://some-workspace/tmp/jr_1d02d47beccb8f7d2c2e8429bbaa3e143f90fcb7837373fb565abdade4a8c845_audit/*.csv

Reverting back to old version s3fs==2024.2.0 works. Note pandas==2.1.4.

[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venv-callsn9i4utg/script.py", line 48, in generate_partition_sql
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     df = pd.read_csv(
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -          ^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/home/***/.local/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 948, in read_csv
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     return _read(filepath_or_buffer, kwds)
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/home/***/.local/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 611, in _read
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     parser = TextFileReader(filepath_or_buffer, **kwds)
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/home/***/.local/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1448, in __init__
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     self._engine = self._make_engine(f, self.engine)
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/home/***/.local/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1705, in _make_engine
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     self.handles = get_handle(
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -                    ^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/home/***/.local/lib/python3.11/site-packages/pandas/io/common.py", line 718, in get_handle
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     ioargs = _get_filepath_or_buffer(
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -              ^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/home/***/.local/lib/python3.11/site-packages/pandas/io/common.py", line 420, in _get_filepath_or_buffer
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     ).open()
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -       ^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/fsspec/core.py", line 135, in open
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     return self.__enter__()
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -            ^^^^^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/fsspec/core.py", line 103, in __enter__
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     f = self.fs.open(self.path, mode=mode)
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/fsspec/spec.py", line 1293, in open
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     f = self._open(
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -         ^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/s3fs/core.py", line 685, in _open
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     return S3File(
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -            ^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/s3fs/core.py", line 2179, in __init__
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     super().__init__(
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/fsspec/spec.py", line 1651, in __init__
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     self.size = self.details["size"]
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -                 ^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/fsspec/spec.py", line 1664, in details
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     self._details = self.fs.info(self.path)
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -                     ^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     return sync(self.loop, func, *args, **kwargs)
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/fsspec/asyn.py", line 103, in sync
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     raise return_result
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/fsspec/asyn.py", line 56, in _runner
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     result[0] = await coro
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -                 ^^^^^^^^^^
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -   File "/tmp/venvogubdfca/lib/python3.11/site-packages/s3fs/core.py", line 1418, in _info
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO -     raise FileNotFoundError(path)
[2024-03-17, 02:39:41 +07] {process_utils.py:190} INFO - FileNotFoundError: some-workspace/tmp/jr_1d02d47beccb8f7d2c2e8429bbaa3e143f90fcb7837373fb565abdade4a8c845_audit/*.csv
[2024-03-17, 02:39:41 +07] {taskinstance.py:2698} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 433, in _execute_task
    result = execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/decorators/base.py", line 241, in execute
    return_value = super().execute(context)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 400, in execute
    return super().execute(context=serializable_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 199, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 716, in execute_callable
    result = self._execute_python_callable_in_subprocess(python_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/operators/python.py", line 471, in _execute_python_callable_in_subprocess
    raise AirflowException(error_msg) from None
airflow.exceptions.AirflowException: Process returned non-zero exit status 1.
some-workspace/tmp/jr_1d02d47beccb8f7d2c2e8429bbaa3e143f90fcb7837373fb565abdade4a8c845_audit/*.csv

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the pandas read_csv call with s3fs 2024.3.0 and compare it with 2024.2.0. Start from the s3fs/core.py and fsspec/core.py paths shown in the traceback, then add a regression test for reading the S3 CSV glob and verify the failing version works without changing the expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, pandas, python
Domain
cloud, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.