GoogleCloudPlatform / GoogleCloudPlatform/gsutil

(FEATURE REQUEST) Parquet Metadata Parser

Open
#1,833 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Hi, I am looking to raise a PR for enhancing the stat command with the ability to parse Parquet for collecting row/column stats so initial profiling can be done on data.

As of now, we have to create a BigQuery Table(External) first to be able to do analysis. Otherwise bq cli has to be used or DataCatalog service.

Wouldn't it be good if this feature would be part of gsutil itself? For future expansion, other file formats can be parsed as well.

I already have a parser ready, I am just working on integrating it with gsutil.

Let me know if I can raise a PR!

Contributor guide

Open the contributing guide

Research direction

Start with gsutil's stat command and review the Parquet parser the issue author says is already prepared, then trace how it should be integrated. Done means the command can collect and report Parquet row and column statistics for initial profiling; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, cloud, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.