kernelci / kernelci/kernelci-api

Optimize node collection indexes

Open
#706 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
21
PR merge metrics
No merged PRs in 30d

Description

## Background

The production `node` collection (35.6M docs) has **19 indexes totaling
~11.8 GB — 38% of all data on disk** (`storageSize` 19 GB, `indexSize`
11.8 GB). Every node insert/update pays maintenance cost on all of them,
and they inflate physical backup size and restore time.

`$indexStats` from production (window: ~2 days since pod restart on
2026-07-06) shows roughly half of them unused.

## Finding 1: no `node` indexes are managed by code

`Node` in kernelci-core does not implement `get_indexes()`, so
`db.create_indexes()` creates nothing for the `node` collection. All 18
non-`_id` indexes were created manually on the server with no record of
why. Whatever survives this cleanup should be added to
`Node.get_indexes()` (kernelci-core `api/models.py`) so the index set is
code-managed and reproducible.

## Finding 2: redundant indexes — safe to drop now

- **`data.kernel_revision.commit_1`** (12k ops) — fully covered by the
compound
`data.kernel_revision.commit_1_data.kernel_revision.branch_1_data.kernel_revision.tree_1`
(prefix rule). Drop the single, keep the compound.
- **`created_1_updated_-1`** (0 ops) — `created_1` is its prefix and is
in active use (likely the node purge job). Drop the compound, keep
`created_1`.
- **`id_1`** (0 ops) — the API maps `id` ↔ `_id`; documents likely have
no `id` field. Verify with
`db.node.countDocuments({id: {$exists: true}})` — if 0, drop.

## Finding 3: zero-usage indexes — hide, observe, then drop

Zero ops in the sample window, but the window is too short to catch
weekly/monthly jobs. Hide them (planner stops using them, instantly
reversible with `unhideIndex`), observe 2–4 weeks, then drop what nobody
missed:

- `updated_1`
- `treeid_1` (sparse)
- `owner_1`
- `data.error_code_1`
- `result_1`
- `group_1`
- `data.kernel_revision.branch_1`
- `data.kernel_revision.tree_1`

```js
db.node.hideIndex("")
```

## Indexes confirmed in active use (keep)

`_id_` (4.0M ops), `parent_1` (2.3M), `state_1` (712k), `kind_1` (241k),
`data.kernel_revision.commit_1_..._tree_1` compound,
`processed_by_kcidb_bridge_1_updated_-1`, `name_1`, `created_1`.

## Expected impact

- Estimated 2–4 GB reduction in index footprint (more after observation
phase drops).
- Reduced write amplification on every node insert/update.
- Smaller physical backups and faster restores.

## Plan

1. [ ] Capture per-index sizes:
`db.node.aggregate([{$collStats: {storageStats: {}}}])` →
`indexSizes`
2. [ ] Drop `data.kernel_revision.commit_1`, `created_1_updated_-1`;
verify and drop `id_1`
3. [ ] Hide the 8 zero-usage indexes, re-check `$indexStats` after
2–4 weeks
4. [ ] Drop unused ones
5. [ ] Add surviving indexes to `Node.get_indexes()` in kernelci-core

Issue written with help of Codex AI assistant

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the production index inventory and size query in the issue, then inspect Node.get_indexes() in kernelci-core/api/models.py. Verify the id-field count before dropping id_1, remove or hide the specified indexes, and re-check $indexStats after the observation period. Done means the surviving indexes are code-managed and the planned footprint reduction is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.