lablup / lablup/backend.ai

Clarify (or add) a usage-recalculation trigger for vfolders written directly on the filesystem

Open
#12,833 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Problem

Direct-filesystem writes — by **BAIFS and by the existing tus upload path alike** — do not synchronously update `vfolders.cur_size` / `num_files` / `last_used`. The current source shows no push mechanism a direct-FS writer could call to keep those DB counters current, so they lag reality. (Backend quota enforcement at the quota-scope directory — XFS/NetApp/CephFS — is unaffected; this is only about the DB-tracked counters.)

## Request

Clarification and, if none exists, a small addition:

1. **What is the intended refresh mechanism** for these counters — periodic scan, storage-proxy usage report, something else? Documenting the SLA lets satellite services set correct expectations.
2. **Is there (or can there be) an explicit "recalculate usage for vfolder X" trigger** — an idempotent, on-demand recompute a satellite could optionally call after a batch of writes?

If such a trigger exists or is added, BAIFS would ship an **optional reconciliation hook** to invoke it, keeping the DB counters closer to the filesystem truth without making it a hot-path dependency.

## Rationale

- The counter drift is a pre-existing property of the tus path, not new to BAIFS; a documented SLA + optional trigger benefits both.
- Keeps v1 semantics honest ("DB counters lag; the filesystem is authoritative") while offering a supported path to reconcile.

## Citations (`6d680960d`)

- Direct-FS write counter gap and the v1 SLA: [docs/01 §2 mitigations table](../../01-architecture.md), [docs/07 Q5](../../07-risks-open-questions.md).
- Advisory `max_files`/`max_size` columns not enforced on the file path (parity with tus): [docs/07 Q5](../../07-risks-open-questions.md).

---
_Filed from the BAIFS project: see lablup/BAIFS#6 and `docs/spikes/upstream-asks/05-usage-accounting-trigger.md` for full context and citations._

Contributor guide

Open the contributing guide

Research direction

Read docs/01-architecture.md §2 and docs/07-risks-open-questions.md Q5, then inspect the existing vfolder usage-accounting and tus paths referenced by the issue. Confirm the current refresh mechanism and SLA, and determine whether an idempotent on-demand trigger exists; done means the behavior is documented or the trigger contract is specified and validated by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, databases, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.