fsspec / fsspec/filesystem_spec

Integration with pyodide-http

Open
#1,069 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

https://github.com/koenvo/pyodide-http provides patches to requests and urllib, to enable python stack HTTP calls in the browser. Since several fsspec backends (http, s3, gcs. azure) depend on HTTP, this raises the possibility of fsspec and pydata IO in the browser.

Principle problem:

  • fsspec's backends are async-behind-sync, and use a dedicated IO thread and loop. Perhaps this is analogous to the way pyodide-http uses a webworker. ( https://github.com/koenvo/pyodide-http/issues/11 ). These backends can also be used in async mode, when called from async code, but without file-like objects (since there is no async standard class to duck-type to)
  • CORS (does the s3 API service, for instance, allow global CORS? I can only find info on how to set CORS for the HTTP endpoints of some public bucket)

Alternatively, sync versions of these backends would already be useful, and some existing remote backends are sync and using requests (github, dropbox?)

cc @koenvo @philippjfr

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the HTTP, S3, GCS, and Azure backends, along with the sync and async paths described in the issue. Read the linked pyodide-http issue 11 and inspect the mentioned GitHub and Dropbox backends for existing requests-based implementations. Done requires a defined browser-compatible integration approach, including how IO execution and CORS are handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, web-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.