kernelci / kernelci/kernelci-api

Optimize node collection indexes

Abierto
#706 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
10
Forks
21
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Background

The production `node` collection (35.6M docs) has **19 indexes totaling
~11.8 GB — 38% of all data on disk** (`storageSize` 19 GB, `indexSize`
11.8 GB). Every node insert/update pays maintenance cost on all of them,
and they inflate physical backup size and restore time.

`$indexStats` from production (window: ~2 days since pod restart on
2026-07-06) shows roughly half of them unused.

## Finding 1: no `node` indexes are managed by code

`Node` in kernelci-core does not implement `get_indexes()`, so
`db.create_indexes()` creates nothing for the `node` collection. All 18
non-`_id` indexes were created manually on the server with no record of
why. Whatever survives this cleanup should be added to
`Node.get_indexes()` (kernelci-core `api/models.py`) so the index set is
code-managed and reproducible.

## Finding 2: redundant indexes — safe to drop now

- **`data.kernel_revision.commit_1`** (12k ops) — fully covered by the
compound
`data.kernel_revision.commit_1_data.kernel_revision.branch_1_data.kernel_revision.tree_1`
(prefix rule). Drop the single, keep the compound.
- **`created_1_updated_-1`** (0 ops) — `created_1` is its prefix and is
in active use (likely the node purge job). Drop the compound, keep
`created_1`.
- **`id_1`** (0 ops) — the API maps `id` ↔ `_id`; documents likely have
no `id` field. Verify with
`db.node.countDocuments({id: {$exists: true}})` — if 0, drop.

## Finding 3: zero-usage indexes — hide, observe, then drop

Zero ops in the sample window, but the window is too short to catch
weekly/monthly jobs. Hide them (planner stops using them, instantly
reversible with `unhideIndex`), observe 2–4 weeks, then drop what nobody
missed:

- `updated_1`
- `treeid_1` (sparse)
- `owner_1`
- `data.error_code_1`
- `result_1`
- `group_1`
- `data.kernel_revision.branch_1`
- `data.kernel_revision.tree_1`

```js
db.node.hideIndex("")
```

## Indexes confirmed in active use (keep)

`_id_` (4.0M ops), `parent_1` (2.3M), `state_1` (712k), `kind_1` (241k),
`data.kernel_revision.commit_1_..._tree_1` compound,
`processed_by_kcidb_bridge_1_updated_-1`, `name_1`, `created_1`.

## Expected impact

- Estimated 2–4 GB reduction in index footprint (more after observation
phase drops).
- Reduced write amplification on every node insert/update.
- Smaller physical backups and faster restores.

## Plan

1. [ ] Capture per-index sizes:
`db.node.aggregate([{$collStats: {storageStats: {}}}])` →
`indexSizes`
2. [ ] Drop `data.kernel_revision.commit_1`, `created_1_updated_-1`;
verify and drop `id_1`
3. [ ] Hide the 8 zero-usage indexes, re-check `$indexStats` after
2–4 weeks
4. [ ] Drop unused ones
5. [ ] Add surviving indexes to `Node.get_indexes()` in kernelci-core

Issue written with help of Codex AI assistant

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Start with the production index inventory and size query in the issue, then inspect Node.get_indexes() in kernelci-core/api/models.py. Verify the id-field count before dropping id_1, remove or hide the specified indexes, and re-check $indexStats after the observation period. Done means the surviving indexes are code-managed and the planned footprint reduction is verified.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
mongodb, python
Área
databases
Tipo de issue
Refactorización
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.