Question: Using dask/s3fs with non-standard iterator

Open
#289 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
18/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws, python
Domain
cloud, data

Research direction

No files, tests, or entry points are mentioned. Start by reviewing the dask/s3fs interfaces for loading S3 data into dataframes and determine whether custom iterators are supported; done would require documented guidance or a clearly scoped implementation path for non-standard binary files.

Written by the indexing model from the issue text.

Description

Hi all,

We're hoping to use dask/s3fs with the below use case:

  1. We have many large binary data files stored on S3, which we hope to process
  2. Our aim is to load parts of the data into dask dataframes using s3fs and an iterator

We're able to get our iterator to work when we store the non-standard files locally. But it seems like the ability of dask/s3fs to load data from s3 easily is limited to csv, parquet and json.

Is there a way to use a custom iterator function to load non-standard binary data stored on s3 into a dask dataframe using s3fs? Any examples/guidance would be highly appreciated.

Martin

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fsspec/s3fs

All issues in fsspec/s3fs

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.