eodash / eodash/eodash_catalog

Implement collection update strategy logic (for geoparquet)

Open
#44 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
3
Forks
3
Avg merge
14m
Merged PRs (30d)
5

Description

If we start using geoparquets, we could add something like "update_strategy" to the collections, which could be for example:
static, full, update.
So for example with update we could first fetch the geoparquet from the location we would expect, see if available and what the last entry (end time extent value) is, then only request items after that, filtering for time, adding them to the geoparquet.
That would mean the catalog update could be quite fast only needing fetching the difference to the previous generation.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by locating the collection update and catalog generation logic, then determine how geoparquet availability, end-time extents, filtering, and appending are currently handled. Done means collections support static, full, and update strategies with incremental catalog updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.