developmentseed / developmentseed/virtualizarr-data-pipelines

Cron job for dimension expansion.

Open
#22 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

The currently forward processing pipeline trys to operate in an atomic way. When a new file notification arrives, the `processor` attempts to append it to the store (normally in the time dimension). This can create an issue where multiple simultaneous invocations of the processor can create a conflict where each worker is trying to expand the time dimension.

We should consider an alternative approach for datasets that are updated at a regular cadence. We can maintain a separate cron function that expands the time dimension as a single operation and then each worker can effectively use region writing without conflict.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by tracing the processor entry point and the notification path, then determine how the store expands the time dimension and how a separate cron function would coordinate with workers. Done means the scheduled expansion and concurrent region writes avoid dimension-expansion conflicts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.