lablup / lablup/backend.ai

Add usage tracking and quota enforcement for artifact storage

Open
#8,194 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Motivation

Currently, artifact storage has no usage tracking or capacity limits. When
artifacts are imported, they are stored in storage namespaces without any
visibility into how much space is being consumed or any mechanism to prevent
storage exhaustion.

This creates operational risks:

- Storage can be exhausted without warning
- No visibility into storage utilization per namespace
- No way to enforce capacity planning or cost control

The infrastructure to support this already partially exists:

- `ArtifactRevisionRow.size` tracks individual revision sizes
- `AssociationArtifactsStorageRow` links revisions to storage namespaces
- We just need to aggregate this data and enforce limits

## Objective

Implement usage tracking and quota enforcement at the **Storage Namespace**
level for artifact storage.

Key capabilities:

1. Track current storage usage per namespace (total size, artifact count,
revision count)
1. Configure maximum capacity limits per storage namespace
1. Enforce limits during artifact import operations
1. Provide API endpoints to query usage and manage quotas

## Expected Sub Issues

1. **Add quota configuration to StorageNamespaceRow**
- Add `max_size` field to database schema
- Create Alembic migration
- Update data types and repository layer
1. **Implement usage tracking and quota enforcement**
- Add usage aggregation query (JOIN artifact_revisions with association
table)
- Create quota check service
- Integrate quota validation into artifact import flow
1. **Add API endpoints for usage and quota management**
- GET endpoint to retrieve storage namespace usage statistics
- PATCH endpoint to update storage namespace quota (admin)

JIRA Issue: BA-3989

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.