0xMiden / 0xMiden/note-transport-service

No graceful shutdown (SIGTERM drops in-flight work, streams, telemetry)

Open
#119 1 comment 0 reactions 0 assignees View on GitHub
enhancement production-readiness
Dominant language
Rust
Stars
3
Forks
10
Avg merge
2h 23m
Merged PRs (30d)
4

Description

Severity: critical/high.

### Summary

There is no signal handling anywhere. `node.entrypoint()` (`bin/node/src/main.rs:80-81`) calls into `Server::serve()` (`crates/node/src/node/grpc/mod.rs:126`, not `serve_with_shutdown`), and no `tokio::signal` usage exists. The OTEL providers are never shut down (`crates/node/src/logging.rs` has no `shutdown()` calls).

### Impact

On SIGTERM (k8s eviction, `docker stop`) the process is terminated with default disposition:

- in-flight `SendNote` writes are cut (WAL keeps this crash-safe, but not clean);
- open `StreamNotes` streams are dropped without a GOAWAY drain;
- health is never flipped to NOT_SERVING, so the LB keeps routing during shutdown;
- OTEL spans/metrics buffered since the last export are lost.

### Recommendation

Install a SIGTERM + `ctrl_c` handler that: marks health NOT_SERVING, calls `serve_with_shutdown` with a drain grace period, signals the streamer to stop, cancels the maintenance task, and shuts down the OTEL tracer/meter providers.

---
Part of #114.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.