GoogleCloudPlatform / GoogleCloudPlatform/gsutil
(FEATURE REQUEST) Parquet Metadata Parser
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I am looking to raise a PR for enhancing the stat command with the ability to parse Parquet for collecting row/column stats so initial profiling can be done on data.
As of now, we have to create a BigQuery Table(External) first to be able to do analysis. Otherwise bq cli has to be used or DataCatalog service.
Wouldn't it be good if this feature would be part of gsutil itself? For future expansion, other file formats can be parsed as well.
I already have a parser ready, I am just working on integrating it with gsutil.
Let me know if I can raise a PR!
Contributor guide
Research direction
Start with gsutil's stat command and review the Parquet parser the issue author says is already prepared, then trace how it should be integrated. Done means the command can collect and report Parquet row and column statistics for initial profiling; the issue does not name specific files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, cloud, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100