kernelci / kernelci/kernelci-api
Optimize node collection indexes
- 主要言語
- Python
- スター
- 10
- フォーク
- 21
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
## Background
The production `node` collection (35.6M docs) has **19 indexes totaling
~11.8 GB — 38% of all data on disk** (`storageSize` 19 GB, `indexSize`
11.8 GB). Every node insert/update pays maintenance cost on all of them,
and they inflate physical backup size and restore time.
`$indexStats` from production (window: ~2 days since pod restart on
2026-07-06) shows roughly half of them unused.
## Finding 1: no `node` indexes are managed by code
`Node` in kernelci-core does not implement `get_indexes()`, so
`db.create_indexes()` creates nothing for the `node` collection. All 18
non-`_id` indexes were created manually on the server with no record of
why. Whatever survives this cleanup should be added to
`Node.get_indexes()` (kernelci-core `api/models.py`) so the index set is
code-managed and reproducible.
## Finding 2: redundant indexes — safe to drop now
- **`data.kernel_revision.commit_1`** (12k ops) — fully covered by the
compound
`data.kernel_revision.commit_1_data.kernel_revision.branch_1_data.kernel_revision.tree_1`
(prefix rule). Drop the single, keep the compound.
- **`created_1_updated_-1`** (0 ops) — `created_1` is its prefix and is
in active use (likely the node purge job). Drop the compound, keep
`created_1`.
- **`id_1`** (0 ops) — the API maps `id` ↔ `_id`; documents likely have
no `id` field. Verify with
`db.node.countDocuments({id: {$exists: true}})` — if 0, drop.
## Finding 3: zero-usage indexes — hide, observe, then drop
Zero ops in the sample window, but the window is too short to catch
weekly/monthly jobs. Hide them (planner stops using them, instantly
reversible with `unhideIndex`), observe 2–4 weeks, then drop what nobody
missed:
- `updated_1`
- `treeid_1` (sparse)
- `owner_1`
- `data.error_code_1`
- `result_1`
- `group_1`
- `data.kernel_revision.branch_1`
- `data.kernel_revision.tree_1`
```js
db.node.hideIndex("")
```
## Indexes confirmed in active use (keep)
`_id_` (4.0M ops), `parent_1` (2.3M), `state_1` (712k), `kind_1` (241k),
`data.kernel_revision.commit_1_..._tree_1` compound,
`processed_by_kcidb_bridge_1_updated_-1`, `name_1`, `created_1`.
## Expected impact
- Estimated 2–4 GB reduction in index footprint (more after observation
phase drops).
- Reduced write amplification on every node insert/update.
- Smaller physical backups and faster restores.
## Plan
1. [ ] Capture per-index sizes:
`db.node.aggregate([{$collStats: {storageStats: {}}}])` →
`indexSizes`
2. [ ] Drop `data.kernel_revision.commit_1`, `created_1_updated_-1`;
verify and drop `id_1`
3. [ ] Hide the 8 zero-usage indexes, re-check `$indexStats` after
2–4 weeks
4. [ ] Drop unused ones
5. [ ] Add surviving indexes to `Node.get_indexes()` in kernelci-core
Issue written with help of Codex AI assistant
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
Start with the production index inventory and size query in the issue, then inspect Node.get_indexes() in kernelci-core/api/models.py. Verify the id-field count before dropping id_1, remove or hide the specified indexes, and re-check $indexStats after the observation period. Done means the surviving indexes are code-managed and the planned footprint reduction is verified.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- mongodb, python
- 領域
- databases
- issue の種類
- リファクタリング
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 静か
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 45/100