lablup / lablup/backend.ai

VFolder abstraction for object storage (S3)

Open
#665 0 comments 0 reactions 1 assignee Assigned to @jopemachine View on GitHub
RoadMap
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Currently we are supporting object storage (Amazon S3 or compatible ones) by allowing users to install S3-compatible client libraries inside containers and images and let their workloads connect to the external object storage services.

There are increasing demands on "managed" object storage abstraction both from customers and storage vendors. In Backend.AI, a managed storage space is abstracted as a vfolder.

## Potential Mapping Designs

- Bucket-to-VFolder
- Subdirectory-to-VFolder

For simplicity, we’d choose the first one: bucket-to-vfolder.

## Component Extensions

### Storage Proxy

- Let’s treat the connection configuration to a specific object storage service as a storage-proxy backend.
- Backend type: `s3`-compatible ones (including MinIO), …
- Each volume configuration includes the endpoint and the service credentials (like API keys).
- cf) Each vfolder is mapped to a specific storage-proxy volume via its `host` field.

### Manager

- Depending on the storage backend type of a vfolder, we need to pass the object storage endpoint & credentials to the agent when creating sessions.
- Similarly to the unmanaged vfolders ([https://lablup.atlassian.net/browse/BA-114](https://lablup.atlassian.net/browse/BA-114)), we need to apply specialized lifecycle implementations. e.g.:
- _Decouple_ actual filesystem-level creation/deletion of vfolders.
- _Allow_ users to register/deregister vfolder entries assuming that the original bucket’s lifecycle is managed by the object-storage solution.

### Agent

- Before creating a container, the agent lets `s3fs` mount the bucket into the local filesystem and bind-mount it into the container.
- After destroying a container, the agents unmounts the bucket.
- We need to keep track of the references to a specific bucket to prevent duplicate mounts and premature unmounts when there are multiple containers using the same bucket.
- We may need to handle potential system instability due to frequent mount/unmount in the filesystem.

## Technical Considerations

- How to implement **read-only** mounts?
- In the storage-proxy, we need to keep mounts of **ALL** registered buckets to implement managed vfolder interaction APIs. Could this incur too much burden to the storage-proxy nodes?

JIRA Issue: BA-255

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.