lablup / lablup/backend.ai

Add the storage volume, backend and mount schema

Open
#14,219 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Add the tables that BEP-1078 defines, together with the virtual folder reference to a storage volume. This change is additive: no legacy column is dropped here, and no behaviour changes until the heartbeat and mount resolution work lands.

```sql
storage_backends
id GUID(StorageBackendID) PK
name String(64) UNIQUE, NOT NULL -- heartbeat: falls back to `type` when absent
type String(64) NOT NULL -- vfs, xfs, purestorage, weka, netapp, ...
connection_info PydanticColumn(StorageBackendConnection) NULL -- administrator API only
credential SecretColumn("storage_backend_credential") NULL -- administrator API only
supports_vfolder Boolean NOT NULL, default false
supports_metric Boolean NOT NULL, default false
supports_quota Boolean NOT NULL, default false
supports_fast_fs_size Boolean NOT NULL, default false
supports_fast_scan Boolean NOT NULL, default false
supports_fast_size Boolean NOT NULL, default false
created_at / updated_at
-- no status column: an appliance has no overall verdict

service_storage_backends
id GUID PK
service_catalog_id GUID(ServiceCatalogID) FK -> service_catalog.id, NOT NULL
storage_backend_id GUID(StorageBackendID) FK -> storage_backends.id, NOT NULL
status StrEnumType(ServiceStorageStatus) NOT NULL, default 'unknown'
created_at / updated_at
UNIQUE (service_catalog_id, storage_backend_id)

storage_volumes
id GUID(StorageVolumeID) PK
name String(64) UNIQUE, NOT NULL -- the identity; declared in every service that mounts it
storage_backend_id GUID(StorageBackendID) FK -> storage_backends.id, NOT NULL
created_at / updated_at
-- no path and no status: both belong to the service relationship

service_storage_volumes
id GUID PK
service_catalog_id GUID(ServiceCatalogID) FK -> service_catalog.id, NOT NULL
storage_volume_id GUID(StorageVolumeID) FK -> storage_volumes.id, NOT NULL
mount_path String NOT NULL -- this service's own mount path
status StrEnumType(ServiceStorageStatus) NOT NULL, default 'unknown'
created_at / updated_at
UNIQUE (service_catalog_id, storage_volume_id)

resource_group_storage_volumes
id GUID PK
scaling_group_id GUID(ResourceGroupID) FK -> scaling_groups.id, NOT NULL
storage_volume_id GUID(StorageVolumeID) FK -> storage_volumes.id, NOT NULL
enabled Boolean NOT NULL, default true
created_at / updated_at
UNIQUE (scaling_group_id, storage_volume_id)

vfolder_mounts
id GUID PK
vfolder_id GUID(VFolderID) FK -> vfolders.id, NOT NULL
session_id GUID(SessionID) FK -> sessions.id, NOT NULL -- becomes a kernel group reference later
subpath String NOT NULL, default '.'
dst_path String NOT NULL
permission StrEnumType(MountPermission) NOT NULL
created_at / updated_at
UNIQUE (session_id, vfolder_id, dst_path, subpath)
UNIQUE (session_id, dst_path)

vfolders
+ storage_volume_id GUID(StorageVolumeID) FK -> storage_volumes.id, NULL

ServiceStorageStatus = unknown | healthy | unhealthy | detached
'detached' marks a relationship the service no longer reports. Rows are never deleted.
```

Notes:

- A storage backend carries no status. Only the services that mount its volumes can reach it, each over its own network path, so the observation lives on the service relationship and there is no aggregate verdict.
- A storage volume carries neither a path nor a status. The path differs per service, and a volume that no service reports is not unhealthy but simply unreachable.
- Nothing is hard-deleted. A relationship the service stops reporting is marked detached, so a returning service reattaches and the record of what was mounted where survives.
- Built-in backend types are seeded when the schema is created, following the resource slot type precedent, so an existing single-node installation has a working vfs backend before any heartbeat arrives.
- scaling_groups has a unique id column alongside its name primary key; the resource group reference targets that id.
- The virtual folder reference is nullable here. It becomes required for every folder not in a terminal deletion state, enforced by a check constraint, once the host string is retired.

Design: BEP-1078 (Storage Proxy Enhancement).

JIRA Issue: BA-7653

Contributor guide

Open the contributing guide

Research direction

Start by locating the schema or migration entry points and reviewing the existing resource slot type precedent mentioned in the issue. Compare the implementation with the BEP-1078 table definitions, relationships, unique constraints, defaults, and seed requirements. Done means the additive schema is represented without dropping legacy columns or changing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.