lablup / lablup/backend.ai

Bound the size of an app config fragment

Open
#13,730 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Objective

Put an upper bound on how large a single app config fragment may be. Today no layer bounds it, so any principal allowed to write a fragment can store a document of practically unlimited size. Decide the limit, enforce it on every write path, and reject oversize documents with a typed error.

## Background

- `AppConfigFragmentRow.config` is a plain `JSONB` column with no check constraint (`manager/models/app_config_fragment/row.py`). Postgres stops only at the ~1 GB field ceiling.
- `AppConfigFragmentUpsertItem.config` is typed `dict[str, Any]` with no `Field()` constraint (`common/dto/manager/v2/app_config_fragment/request.py`), and `AppConfigFragmentService.bulk_upsert` adds no size check of its own.
- The bulk upsert inputs (`ScopedUpsertAppConfigFragmentsInput.items`, `MyUpsertAppConfigFragmentsInput.items`) declare `min_length=1` but no maximum, so one request may also carry an unbounded number of items.
- The `my` upsert lets an ordinary user write fragments at their own user scope, so the growth path is reachable without any admin role.
- Every app config read merges the fragments for the requested names (`AppConfigService.get_app_configs`), so an oversize fragment is paid for on each read, not just once at write time.

## Scope

- Decide the limit and how it is measured (serialized UTF-8 JSON bytes is the obvious unit).
- Enforce it in the service layer so GraphQL and REST v2 are both covered, and raise a `BackendAIError` subclass added to `manager/errors/app_config.py` rather than a bare validation error.
- Keep the per-item partial-success contract from BA-7250: in a bulk upsert an oversize item fails on its own and does not sink the rest of the batch.
- Decide the maximum item count per bulk upsert request, or split it off explicitly if it is not taken here.
- State the limit in the DTO field description so it shows up in the generated API reference.

## Open questions

- The value itself is undecided — the candidates are on the order of 64 KiB / 256 KiB / 1 MiB per fragment.
- Fixed constant, or configurable through the manager config? A per-config-name limit carried on the definition or the allow-list entry is a third option.
- Whether the merged result of all fragments for one config name needs its own cap on top of the per-fragment one.

## Acceptance Criteria

- A per-fragment size limit is chosen, documented, and enforced on both the scoped and the `my` upsert paths.
- An oversize fragment is rejected with a typed app config error carrying a distinguishable error code, not a generic 400.
- In a bulk upsert, an oversize item rejects the whole batch. Per-item partial reporting waits until the repository supports partial success — the upsert's failed list is never populated today.
- A decision on the per-request item count limit is recorded — implemented, or deferred to a linked issue.
- Tests cover a document just under the limit (accepted), one just over (rejected), and the mixed bulk batch.

## Related

- BA-7230 — App Config follow-up issues (parent epic).
- BA-7250 — Support partial success in bulk app config fragment upsert.

JIRA Issue: BA-7345

Contributor guide

Open the contributing guide

Research direction

Start by reading AppConfigFragmentService.bulk_upsert, manager/models/app_config_fragment/row.py, common/dto/manager/v2/app_config_fragment/request.py, and manager/errors/app_config.py, then trace the scoped and my upsert inputs. Resolve the limit, item-count decision, and conflicting partial-success criteria before implementing. Done means both paths enforce the documented limit, return the typed error, and have boundary and mixed-batch tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
api, backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.