Normalize kernel groups into a table
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
Kernel groups are resolved at enqueue time and stored inside the SessionRow.options JSONB, so nothing can reference a group by key. Normalize them into a table so the rows that belong to a group can point at it.
### Current state
- KernelGroup (manager/data/session/options.py) carries role, replica_count, execution_spec, failure_policy and depends_on_roles.
- SessionStoredOptions writes the resolved list into SessionRow.options JSONB. There is no table, so a group has no id.
- ExpandKernelGroupsRule flattens one group into replica_count kernels, assigning cluster_role = role, a 1-based cluster_idx within the role, cluster_hostname = f"{role}{cluster_idx}" and a session-wide local_rank.
- A group is the unit a session's layout is actually declared in: the v2 create API takes SessionOptionsInput.kernel_groups, each with its own optional execution_spec.
### Proposed schema
```
kernel_groups
id GUID(KernelGroupID) PK
session_id GUID(SessionID) FK -> sessions.id, NOT NULL
role String(64) NOT NULL -- cluster_role of every replica
replica_count Integer NOT NULL
failure_policy StrEnumType(FailurePolicy) NOT NULL
depends_on_roles JSONB NOT NULL, default '[]'
created_at / updated_at
UNIQUE (session_id, role)
```
execution_spec stays in the options JSONB for now: it is a large nested value with no reader that queries into it, and normalizing it is a separate decision.
### Scope
- Add the kernel_groups table and its Row class.
- Backfill from SessionRow.options JSONB for existing sessions.
- Write the rows where the session is created, and read groups from the table rather than the JSONB where a key is needed.
- Keep the JSONB copy until the readers are moved; retiring it is a follow-up.
### Why this comes first
BA-7653 added session_vfolder_mounts keyed by (session_id, dst_path). That cannot represent a valid session: mounts are resolved per role, not per session — SchedulerRepository._resolve_vfolder_mounts_by_role returns dict[role, tuple[VFolderMount, ...]] and every replica of a role copies the same tuple. Destination-path uniqueness is deliberately scoped per kernel as well (MountNameValidationRule), because two kernels may mount the same vfolder at the same path.
Once kernel groups have a key, session_vfolder_mounts becomes kernel_group_vfolder_mounts, keyed by the group, with the destination-path uniqueness scoped to the group. That re-key is tracked separately.
JIRA Issue: BA-7663
Contributor guide
Research direction
Start with manager/data/session/options.py, SessionRow, and ExpandKernelGroupsRule to trace how groups are resolved and stored, then inspect the session-creation path and SchedulerRepository._resolve_vfolder_mounts_by_role. Done means adding the kernel_groups table and Row class, backfilling existing JSONB data, writing groups on creation, and reading them where a group key is needed while retaining the JSONB copy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100