developmentseed / developmentseed/virtualizarr-data-pipelines

Add support for S3 Inventory report Parquet.

Open
#21 0 comments 0 reactions 1 assignee Claimed by @sharkinsspatial View on GitHub
Dominant language
Python
Stars
8
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

To experiment and validate with the partitioned backfill processing I initially only added support for file keys in a `json` array. For large inventory files, Lambda memory may not be sufficient for loading and parsing the list. Ideally we should add support for accessing and parsing Parquet hive partitions for actual inventory files or perhaps even supporting streaming Athena queries.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.