fsspec / fsspec/filesystem_spec
What filesystem to use with parquet files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
This might be a naive question but I have spent a bit of time trying to figure it out and haven't made much progress.
I'm trying to do this workflow for a parquet file:
import fsspec
fs = fsspec.filesystem().open(path_to_file)
This sort of workflow without specifying a protocol finds that the parquet file is a directory and returns IsADirectory exception. So I am trying to figure out which protocol to use. Looking through the docs, two built-in implementations mention parquet files, but they both seem aimed at kerchunk files specifically. I'm not sure if this means I can use them for other uses or not? I tried with protocol="reference" and then I wasn't sure what to use for fo. I am using a local parquet file and I used that for fo, something like this:
fs = fsspec.filesystem("reference", fo=path_to_file).open(path_to_file)
but then it couldn't find my file, though it is sitting in the same directory and I had just given the file name in "path_to_file". I am using local files now but in general wouldn't always be.
Am I taking the wrong approach altogether? Any idea for how to approach this? Thanks.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository file or test is named. Start with the fsspec.filesystem() entry point and the reference protocol's fo argument, then reproduce the local parquet example. Done means documenting the correct approach for local and non-local parquet files, or clearly explaining that no code change is needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100