lablup / lablup/backend.ai

Migrate storage configuration from etcd to the database

Open
#14,222 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Once the normalized storage tables exist and services declare their backends and volumes, move the existing data off etcd and retire what it leaves behind.

Scope:

- Move the existing configuration into the database through a manager CLI command rather than an Alembic migration, so that it can be rehearsed with a dry run and repeated. Volume and backend records are pulled from the running storage proxies, because etcd holds no volume data. The resource group to volume table is seeded with every group and every volume, since resource groups place no restriction on volumes today and without the seed no session could mount anything afterwards.
- Retire the etcd volumes configuration schema, its loader, and the helpers that parse the proxy and volume names out of the virtual folder host string.
- Enforce the storage volume reference on virtual folders. The column is added nullable with the schema; once the migration has run it can be constrained, exempting unmanaged folders, which point at a raw path rather than a volume, and folders in a terminal deletion state.

Design: BEP-1078 (Storage Proxy Enhancement).

JIRA Issue: BA-7655

Contributor guide

Open the contributing guide

Research direction

Start with BEP-1078, the manager CLI, and the existing etcd volumes configuration loader. Trace the normalized storage tables, running storage proxy data, virtual-folder schema, and helpers that parse proxy and volume names. Done means a repeatable dry-run migration seeds the database correctly, retires the old etcd path, and enforces the volume reference for eligible folders.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli, database
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.