apache / apache/parquet-java

parquet-cli rewrite option

Open
#2,981 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

### Describe the usage question you have. Please include as many useful details as possible.

Hi ,

is it possible to read directly from a gcs bucket to prune a column
like rewrite -i gs:/sourcebbucket/part-00549.parquet -o gs://targetbucket/newdata/dd --prune-columns col4

i am getting error
java.lang.RuntimeException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "gs"

### Component(s)

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the parquet-cli rewrite command and reproduce the reported gs:// input and output failure. Trace how rewrite resolves filesystem schemes, then check the Hadoop filesystem setup used by the CLI. Done means the requested GCS prune-columns workflow is supported or its limitation is clearly documented, with the error behavior covered by a test if the project provides one.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cli, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.