kernelci / kernelci/kernelci-api
Optimize node collection indexes
- Langage dominant
- Python
- Étoiles
- 10
- Forks
- 21
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
## Background
The production `node` collection (35.6M docs) has **19 indexes totaling
~11.8 GB — 38% of all data on disk** (`storageSize` 19 GB, `indexSize`
11.8 GB). Every node insert/update pays maintenance cost on all of them,
and they inflate physical backup size and restore time.
`$indexStats` from production (window: ~2 days since pod restart on
2026-07-06) shows roughly half of them unused.
## Finding 1: no `node` indexes are managed by code
`Node` in kernelci-core does not implement `get_indexes()`, so
`db.create_indexes()` creates nothing for the `node` collection. All 18
non-`_id` indexes were created manually on the server with no record of
why. Whatever survives this cleanup should be added to
`Node.get_indexes()` (kernelci-core `api/models.py`) so the index set is
code-managed and reproducible.
## Finding 2: redundant indexes — safe to drop now
- **`data.kernel_revision.commit_1`** (12k ops) — fully covered by the
compound
`data.kernel_revision.commit_1_data.kernel_revision.branch_1_data.kernel_revision.tree_1`
(prefix rule). Drop the single, keep the compound.
- **`created_1_updated_-1`** (0 ops) — `created_1` is its prefix and is
in active use (likely the node purge job). Drop the compound, keep
`created_1`.
- **`id_1`** (0 ops) — the API maps `id` ↔ `_id`; documents likely have
no `id` field. Verify with
`db.node.countDocuments({id: {$exists: true}})` — if 0, drop.
## Finding 3: zero-usage indexes — hide, observe, then drop
Zero ops in the sample window, but the window is too short to catch
weekly/monthly jobs. Hide them (planner stops using them, instantly
reversible with `unhideIndex`), observe 2–4 weeks, then drop what nobody
missed:
- `updated_1`
- `treeid_1` (sparse)
- `owner_1`
- `data.error_code_1`
- `result_1`
- `group_1`
- `data.kernel_revision.branch_1`
- `data.kernel_revision.tree_1`
```js
db.node.hideIndex("")
```
## Indexes confirmed in active use (keep)
`_id_` (4.0M ops), `parent_1` (2.3M), `state_1` (712k), `kind_1` (241k),
`data.kernel_revision.commit_1_..._tree_1` compound,
`processed_by_kcidb_bridge_1_updated_-1`, `name_1`, `created_1`.
## Expected impact
- Estimated 2–4 GB reduction in index footprint (more after observation
phase drops).
- Reduced write amplification on every node insert/update.
- Smaller physical backups and faster restores.
## Plan
1. [ ] Capture per-index sizes:
`db.node.aggregate([{$collStats: {storageStats: {}}}])` →
`indexSizes`
2. [ ] Drop `data.kernel_revision.commit_1`, `created_1_updated_-1`;
verify and drop `id_1`
3. [ ] Hide the 8 zero-usage indexes, re-check `$indexStats` after
2–4 weeks
4. [ ] Drop unused ones
5. [ ] Add surviving indexes to `Node.get_indexes()` in kernelci-core
Issue written with help of Codex AI assistant
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Start with the production index inventory and size query in the issue, then inspect Node.get_indexes() in kernelci-core/api/models.py. Verify the id-field count before dropping id_1, remove or hide the specified indexes, and re-check $indexStats after the observation period. Done means the surviving indexes are code-managed and the planned footprint reduction is verified.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- mongodb, python
- Domaine
- databases
- Type d'issue
- Refactorisation
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- Plutôt claire
- Accessibilité débutants
- 45/100