lablup / lablup/backend.ai

Implement periodic configuration sync from Valkey cache in storage proxy

Open
#6,945 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Overview

Implement a background service in the storage proxy that periodically reads configuration data from Valkey cache and applies changes to the runtime configuration without requiring restarts.

## Context

Once the cache_source views and Valkey clients are implemented, the storage proxy needs a mechanism to:

- Periodically poll configuration changes from Valkey
- Detect changes in registry and storage configurations
- Apply the changes to the running storage proxy instance
- Handle configuration updates gracefully without service interruption

This eliminates the need for manual TOML file updates and storage proxy restarts when configurations change in the manager's database.

## Implementation Tasks

- Design configuration sync service with configurable polling interval
- Implement change detection mechanism (compare cached vs current config)
- Create configuration reload logic for each registry/storage type:
- Artifact registries (`valkey_artifact_registries`)
- Artifact storage (`valkey_artifact_storage`)
- HuggingFace registry (`huggingface_registry`)
- Reservoir registry (`reservoir_registry`)
- Object storage (`object_storage`)
- VFS storage (`vfs_storage`)
- Add transaction safety for configuration updates
- Implement rollback mechanism on configuration validation failures
- Add metrics and logging for sync operations
- Handle edge cases (Valkey unavailable, partial updates, concurrent changes)

## Acceptance Criteria

- Storage proxy polls Valkey at configured intervals (default: 60 seconds)
- Configuration changes in manager DB are reflected in storage proxy within one sync cycle
- No service interruption occurs during configuration updates
- Failed configuration updates are logged with proper error messages
- Metrics track sync operation success/failure rates
- Configuration sync can be disabled via configuration flag for testing
- Unit tests cover change detection and reload logic
- Integration tests validate end-to-end synchronization from DB to storage proxy

JIRA Issue: BA-3155

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.