developmentseed / developmentseed/virtualizarr-data-pipelines
Add support for S3 Inventory report Parquet.
Open
- Dominant language
- Python
- Stars
- 8
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
To experiment and validate with the partitioned backfill processing I initially only added support for file keys in a `json` array. For large inventory files, Lambda memory may not be sufficient for loading and parsing the list. Ideally we should add support for accessing and parsing Parquet hive partitions for actual inventory files or perhaps even supporting streaming Athena queries.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.