apache / apache/cloudstack

prometheus improvements

未關閉
#13,667 4 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
type:technical-debt
主要語言
Java
星號
3.1k
分支
1.4k
平均合併
7 天 14 小時
30 天內合併 PR
31

描述

### problem

1. Give the exporter's HttpServer an explicit bounded executor (httpServer.setExecutor(Executors.newFixedThreadPool(2))) so one slow scrape can't serialize/queue all others.
2. Add a short TTL/in-flight guard around updateMetrics() (e.g., skip recompute if last run was < N seconds ago, or synchronize so concurrent scrapes share one in-progress computation) so scrape frequency can never multiply backend load.
3. Instrument: log/measure updateMetrics() wall-clock time so the reporter (and CI) can confirm which sub-metric collector is actually slow and verify the fix closes the growth.

additional comments:

1. Stale dynamic config (CONFIRMED) — capacity.calculate.workers is a runtime-dynamic setting, but the new shared executor only reads it once at first creation; live changes are silently ignored until a restart.
2. Swallowed capacity-recalculation abort (CONFIRMED per the extra verify pass) — shutdown racing an in-flight recalculation throws RejectedExecutionException, caught by the blanket catch(Throwable) in recalculateCapacity(), silently skipping storage/IP/VLAN updates for that cycle.
3. Unsynchronized race on _capacityExecutorService (CONFIRMED per the extra verify pass) — can leak a freshly-recreated pool that's never shut down again.
4. Shared fixed-size pool serializes previously-independent callers (PLAUSIBLE) — rolling-maintenance host-drain gating can now queue behind the hourly timer or API-triggered recalculations.
5. Pool no longer bounded to actual task count, so it can stay oversized/stale relative to fleet size (efficiency).
6. Bundling this executor-lifecycle rewrite into what the reported bug (#13586) only needed a one-line fix for (altitude/scope creep).
7. Inconsistent lazy-vs-eager thread-pool lifecycle pattern within the same class (reuse/convention).
8. Minor: the synchronized getter is called per-loop-iteration instead of hoisted once (efficiency).

貢獻指南

開啟貢獻指南

研究方向

定位 exporter HttpServer、updateMetrics()、recalculateCapacity() 和 _capacityExecutorService;先閱讀它們的執行器生命週期以及 capacity.calculate.workers 的處理方式。追蹤並發 scrape、關閉競爭,以及 rolling-maintenance、API 和計時器呼叫方。完成意味著並行性、動態設定、abort 處理和時間測量 instrumentation 都是明確且可驗證的,同時不會靜默遺失重新計算。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
java, prometheus
領域
infrastructure, observability
Issue 類型
重構
難度
5/5
預估耗時
一週以上
活躍度
冷清
描述清晰度
需要釐清
新手友好度
28/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。