lablup / lablup/backend.ai

Migrate storage configuration from etcd to the database via a manager CLI command

Open
#14,268 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Move the existing storage configuration into the normalized tables. This runs as a manager CLI subcommand rather than an Alembic migration, so that it can be rehearsed with a dry run and repeated; Alembic only creates and drops schema.

- Pull volume and backend records from the running storage proxies. etcd holds no volume or backend data, so the proxies are the only source; doing it here means an operator does not have to wait for every proxy to be upgraded to the new heartbeat.
- Split the virtual folder host string into a storage volume reference. A host naming a volume that no longer exists still gets a record, so the reference stays valid.
- Convert the default host setting to a storage volume reference.
- Seed the resource group to volume table with every resource group and every volume. Resource groups place no restriction on volumes today, so seeding preserves current behaviour; without it no session could mount anything after the migration.
- Be idempotent and offer a dry run. Rolling back is not covered by an Alembic downgrade, so the command must be safe to re-run and must leave the etcd data intact.

Design: BEP-1078 (Storage Proxy Enhancement).

JIRA Issue: BA-7693

Contributor guide

Open the contributing guide

Research direction

Read BEP-1078 and locate the manager CLI entry point and normalized storage tables. Trace how running storage proxies expose volume and backend records, then define completion around an idempotent, dry-run-capable command that preserves etcd data and seeds all required references.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.