eodash / eodash/eodash_catalog
Implement collection update strategy logic (for geoparquet)
- Dominant language
- Python
- Stars
- 3
- Forks
- 3
- Avg merge
- 14m
- Merged PRs (30d)
- 5
Description
If we start using geoparquets, we could add something like "update_strategy" to the collections, which could be for example:
static, full, update.
So for example with update we could first fetch the geoparquet from the location we would expect, see if available and what the last entry (end time extent value) is, then only request items after that, filtering for time, adding them to the geoparquet.
That would mean the catalog update could be quite fast only needing fetching the difference to the previous generation.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by locating the collection update and catalog generation logic, then determine how geoparquet availability, end-time extents, filtering, and appending are currently handled. Done means collections support static, full, and update strategies with incremental catalog updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100