fsspec / fsspec/filesystem_spec

What filesystem to use with parquet files

Open
#1,648 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

This might be a naive question but I have spent a bit of time trying to figure it out and haven't made much progress.

I'm trying to do this workflow for a parquet file:

import fsspec

fs = fsspec.filesystem().open(path_to_file)

This sort of workflow without specifying a protocol finds that the parquet file is a directory and returns IsADirectory exception. So I am trying to figure out which protocol to use. Looking through the docs, two built-in implementations mention parquet files, but they both seem aimed at kerchunk files specifically. I'm not sure if this means I can use them for other uses or not? I tried with protocol="reference" and then I wasn't sure what to use for fo. I am using a local parquet file and I used that for fo, something like this:

fs = fsspec.filesystem("reference", fo=path_to_file).open(path_to_file)

but then it couldn't find my file, though it is sitting in the same directory and I had just given the file name in "path_to_file". I am using local files now but in general wouldn't always be.

Am I taking the wrong approach altogether? Any idea for how to approach this? Thanks.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository file or test is named. Start with the fsspec.filesystem() entry point and the reference protocol's fo argument, then reproduce the local parquet example. Done means documenting the correct approach for local and non-local parquet files, or clearly explaining that no code change is needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.