influxdata / influxdata/influxdb
Expose write path memory pool metrics via `/metrics` endpoint
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
### Problem
As part of separating write path executor work (see [PR](https://github.com/influxdata/influxdb/pull/26455)), we have introduced another metrics registry as when building an executor `iox` expects it to be mapped to a single memory pool and assumes that there's another memory pool configured to same executor when memory pool instrumentation is registered already.
```
2025-06-02T14:49:39.205230Z ERROR panic_logging: Thread panic panic_type="unknown" panic_message="More than one execution pool created: previously existing instrument" panic_file="/home/praveen/.cargo/git/checkouts/influxdb3_core-2ede6fca005e1dcf/fd0e474/iox_query/src/exec.rs" panic_line=281 panic_column=9
thread 'main' panicked at /home/praveen/.cargo/git/checkouts/influxdb3_core-2ede6fca005e1dcf/fd0e474/iox_query/src/exec.rs:281:9:
More than one execution pool created: previously existing instrument
stack backtrace:
0: 0x629f44b5e172 - ::fmt::hc04c8f544ab24d66
1: 0x629f44b8eb63 - core::fmt::write::hfe57b7174b7d8eab
2: 0x629f44b595a3 - std::io::Write::write_fmt::h154385efa8565236
3: 0x629f44b5dfc2 - std::sys::backtrace::BacktraceLock::print::h0c8f24e22f5873a8
4: 0x629f44b5f24c - std::panicking::default_hook::{{closure}}::hd07d57e6a602c8e4
5: 0x629f44b5f04f - std::panicking::default_hook::h63d12f7d95bd91ed
6: 0x629f3fd807db - panic_logging::SendPanicsToTracing::new_inner::{{closure}}::h4f1478e3035af477
7: 0x629f44b5fd43 - std::panicking::rust_panic_with_hook::h33b18b24045abff4
8: 0x629f44b5f9c6 - std::panicking::begin_panic_handler::{{closure}}::hf8313cc2fd0126bc
9: 0x629f44b5e679 - std::sys::backtrace::__rust_end_short_backtrace::h57fe07c8aea5c98a
10: 0x629f44b5f68d - __rustc[95feac21a9532783]::rust_begin_unwind
11: 0x629f44b8bac0 - core::panicking::panic_fmt::hd54fb667be51beea
12: 0x629f414b6cb7 - iox_query::exec::Executor::new_with_config_and_executor::h3ef1059edcb25ade
13: 0x629f3f99fd9c - influxdb3::commands::serve::command::{{closure}}::h2cdf5ca9df83df25
14: 0x629f3f9b6fcb - influxdb3::main::{{closure}}::hc953cfc298ca6770
15: 0x629f3f987b39 - tokio::runtime::park::CachedParkThread::block_on::h51b18ac33f8a0e4d
16: 0x629f3fb2a7bf - tokio::runtime::runtime::Runtime::block_on::h9eb33b87acb6fa53
17: 0x629f3fc20d55 - influxdb3::main::h75a268e75e689bc6
18: 0x629f3fcbb256 - std::sys::backtrace::__rust_begin_short_backtrace::h5b4e77177edb3cca
19: 0x629f3fab4321 - std::rt::lang_start::{{closure}}::hc69eb1d94c6de306
20: 0x629f44b4e080 - std::rt::lang_start_internal::h418648f91f5be3a1
21: 0x629f3fc3b19d - main
22: 0x7ed71c33d488 -
23: 0x7ed71c33d54c - __libc_start_main
24: 0x629f3f95b325 - _start
25: 0x0 -
2025-06-02T14:49:39.295724Z WARN executor: DedicatedExecutor dropped without calling shutdown()
2025-06-02T14:49:39.296308Z WARN executor: DedicatedExecutor dropped without calling shutdown()
```
However in this case, we've created a separate executor with it's own memory pool, but reusing the registry. To avoid running into the panic, a separate registry is created and passed in, although the metrics are never exposed.
### Options
- We can change the `/metrics` endpoint behaviour to either expose the write path memory pool metrics as well
- extra query param to switch (i.e `/metrics?pool=write`) the registry
- see if it's possible to pass in the same reporter but have a different name for the metrics when reporting
- Create a namespace (possibly) such that write path executor pool doesn't conflict with query path. This will mean there's only a single metrics registry that we could reuse.
- Make a change to `iox_query` ([this](https://github.com/influxdata/influxdb3_core/blob/fd0e474a6c0af5ba867399d753f5df18f59907cb/iox_query/src/exec.rs#L203-L206) function) upstream such that it allows an optional name to the datafusion memory pool, so that write path and query path can register different names.
### Solution
TBD
Contributor guide
Research direction
Start by reading the linked iox_query/src/exec.rs function and tracing how the separate write-path registry is passed to the /metrics endpoint. Compare the listed options and determine how write-path memory-pool metrics should be exposed without the executor registry panic; the issue is done when the metrics are available and both executors initialize successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100