fsspec / fsspec/filesystem_spec
Integration with pyodide-http
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
https://github.com/koenvo/pyodide-http provides patches to requests and urllib, to enable python stack HTTP calls in the browser. Since several fsspec backends (http, s3, gcs. azure) depend on HTTP, this raises the possibility of fsspec and pydata IO in the browser.
Principle problem:
- fsspec's backends are async-behind-sync, and use a dedicated IO thread and loop. Perhaps this is analogous to the way pyodide-http uses a webworker. ( https://github.com/koenvo/pyodide-http/issues/11 ). These backends can also be used in async mode, when called from async code, but without file-like objects (since there is no async standard class to duck-type to)
- CORS (does the s3 API service, for instance, allow global CORS? I can only find info on how to set CORS for the HTTP endpoints of some public bucket)
Alternatively, sync versions of these backends would already be useful, and some existing remote backends are sync and using requests (github, dropbox?)
cc @koenvo @philippjfr
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the HTTP, S3, GCS, and Azure backends, along with the sync and async paths described in the issue. Read the linked pyodide-http issue 11 and inspect the mentioned GitHub and Dropbox backends for existing requests-based implementations. Done requires a defined browser-compatible integration approach, including how IO execution and CORS are handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, web-dev
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100