developmentseed / developmentseed/virtualizarr-data-pipelines
Cron job for dimension expansion.
- Dominant language
- Python
- Stars
- 8
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
The currently forward processing pipeline trys to operate in an atomic way. When a new file notification arrives, the `processor` attempts to append it to the store (normally in the time dimension). This can create an issue where multiple simultaneous invocations of the processor can create a conflict where each worker is trying to expand the time dimension.
We should consider an alternative approach for datasets that are updated at a regular cadence. We can maintain a separate cron function that expands the time dimension as a single operation and then each worker can effectively use region writing without conflict.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by tracing the processor entry point and the notification path, then determine how the store expands the time dimension and how a separate cron function would coordinate with workers. Done means the scheduled expansion and concurrent region writes avoid dimension-expansion conflicts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100