lablup / lablup/backend.ai

Normalize kernel groups into a table

Open
#14,232 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Kernel groups are resolved at enqueue time and stored inside the SessionRow.options JSONB, so nothing can reference a group by key. Normalize them into a table so the rows that belong to a group can point at it.

### Current state

- KernelGroup (manager/data/session/options.py) carries role, replica_count, execution_spec, failure_policy and depends_on_roles.
- SessionStoredOptions writes the resolved list into SessionRow.options JSONB. There is no table, so a group has no id.
- ExpandKernelGroupsRule flattens one group into replica_count kernels, assigning cluster_role = role, a 1-based cluster_idx within the role, cluster_hostname = f"{role}{cluster_idx}" and a session-wide local_rank.
- A group is the unit a session's layout is actually declared in: the v2 create API takes SessionOptionsInput.kernel_groups, each with its own optional execution_spec.

### Proposed schema

```
kernel_groups
id GUID(KernelGroupID) PK
session_id GUID(SessionID) FK -> sessions.id, NOT NULL
role String(64) NOT NULL -- cluster_role of every replica
replica_count Integer NOT NULL
failure_policy StrEnumType(FailurePolicy) NOT NULL
depends_on_roles JSONB NOT NULL, default '[]'
created_at / updated_at
UNIQUE (session_id, role)
```

execution_spec stays in the options JSONB for now: it is a large nested value with no reader that queries into it, and normalizing it is a separate decision.

### Scope

- Add the kernel_groups table and its Row class.
- Backfill from SessionRow.options JSONB for existing sessions.
- Write the rows where the session is created, and read groups from the table rather than the JSONB where a key is needed.
- Keep the JSONB copy until the readers are moved; retiring it is a follow-up.

### Why this comes first

BA-7653 added session_vfolder_mounts keyed by (session_id, dst_path). That cannot represent a valid session: mounts are resolved per role, not per session — SchedulerRepository._resolve_vfolder_mounts_by_role returns dict[role, tuple[VFolderMount, ...]] and every replica of a role copies the same tuple. Destination-path uniqueness is deliberately scoped per kernel as well (MountNameValidationRule), because two kernels may mount the same vfolder at the same path.

Once kernel groups have a key, session_vfolder_mounts becomes kernel_group_vfolder_mounts, keyed by the group, with the destination-path uniqueness scoped to the group. That re-key is tracked separately.

JIRA Issue: BA-7663

Contributor guide

Open the contributing guide

Research direction

Start with manager/data/session/options.py, SessionRow, and ExpandKernelGroupsRule to trace how groups are resolved and stored, then inspect the session-creation path and SchedulerRepository._resolve_vfolder_mounts_by_role. Done means adding the kernel_groups table and Row class, backfilling existing JSONB data, writing groups on creation, and reading them where a group key is needed while retaining the JSONB copy.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.