IntersectMBO / IntersectMBO/cardano-node

Support centralised logging and metrics in cardano-testnet via cardano-tracer

Open
#6,490 4 comments 0 reactions 1 assignee Claimed by @isovector View on GitHub
Dominant language
Haskell
Stars
3.2k
Forks
754
Avg merge
4d 20h
Merged PRs (30d)
20

Description

## Summary

Currently, enabling the Prometheus/`PrometheusSimple` backend in `cardano-testnet` is not practical for multi-node testnets because all nodes share the same configuration. This causes port collisions since every node attempts to listen on the same `PrometheusSimple` endpoint (hardcoded to `0.0.0.0:12798`). More broadly, there is no centralised logging — each node logs independently to its own stdout/file.

The relevant code is commented out in `cardano-testnet/src/Testnet/Defaults.hs` (line 306) with a note explaining the limitation and suggesting `cardano-tracer` as the proper solution.

## Problem

- All testnet nodes share a single config, so there is no way to assign unique Prometheus ports per node — only single-node testnets can enable Prometheus without collisions.
- Each node logs independently (Katip file/stdout scribes). There is no unified view of traces across a multi-node testnet.
- Developers debugging or monitoring multi-node testnets lack an easy built-in metrics endpoint and centralised log aggregation.

## Proposed solution: integrate cardano-tracer

`cardano-tracer` is purpose-built for this. It acts as a centralised aggregator that connects to multiple nodes and provides both unified logging and a single Prometheus endpoint with per-node sub-routes.

### Architecture

```
testnet nodes (N) cardano-tracer (1 process)
+------------+ +---------------------------+
| node-spo1 |--\ | Accepts on local socket |
| node-spo2 |-----> forwarder.sock | |
| node-spo3 |--/ | Exposes: |
+------------+ | - Prometheus :3200 |
| - Per-node log dirs |
+---------------------------+
/tracer-logs
/node-spo1/node.json
/node-spo2/node.json
/node-spo3/node.json
```

### Implementation outline

1. **New CLI flag** — add an `--enable-tracer` option to `CardanoTestnetOptions`, following the existing `--enable-grpc` / `RpcSupport` pattern in `Testnet/Start/Types.hs`.

2. **Spawn `cardano-tracer` as an auxiliary process** — following the existing `SubmitApi` pattern in `Testnet/SubmitApi.hs`:
- Generate a tracer config (AcceptAt on a local socket, logging to a per-testnet directory, Prometheus on a free port).
- Spawn `cardano-tracer --config ` before starting nodes.
- Register it for cleanup with the existing `MonadResource` / SIGINT handler infrastructure.

3. **Add tracer socket arg to each node** — in `Testnet/Start/Cardano.hs`, append `--tracer-socket-path-connect ` to each node's CLI args. All nodes connect to the same socket, so no per-node config divergence is needed.

4. **Clean up commented-out code** — the `PrometheusSimple` workaround in `Defaults.hs:306-329` becomes obsolete and can be removed.

### What this enables

- **Centralised logs**: Per-node subdirectories under a single root, with rotation — one place to look at all testnet traces.
- **Single Prometheus endpoint**: Lists all connected nodes at the root, each with its own metrics sub-route.
- **Prometheus service discovery**: `GET /targets` for dynamic scraping configurations.
- **No port collisions**: Only the tracer binds network ports, not individual nodes.
- **Scales to any node count**: `AcceptAt` mode requires zero tracer config changes when adding nodes.

### Files likely affected

- `Testnet/Start/Types.hs` — new option type
- `Parsers/Cardano.hs` — CLI flag parsing
- `Testnet/Start/Cardano.hs` — spawn tracer process, add `--tracer-socket-path-connect` to node args
- `Testnet/Defaults.hs` — tracer config generation, remove commented-out `PrometheusSimple` code

## References

- Comment in source: `cardano-testnet/src/Testnet/Defaults.hs:306`
- [cardano-tracer docs (Cardano Developer Portal)](https://developers.cardano.org/docs/get-started/infrastructure/node/new-tracing-system/cardano-tracer/)
- `cardano-tracer/docs/cardano-tracer.md`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.