microsoft / microsoft/CCF

Logs export / Observability

Open
#7,858 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
876
Forks
260
Avg merge
1d 11h
Merged PRs (30d)
157

Description

Currently logging is expensive in production and hence we need to be careful and 'minimal' in the logs which we do generate.
The idea here is that we want to simplify the logging story using more 'standard-practice' techniques rather than just logging on stdout, and hence also free up logging budget for observability.
The eventual aim is to enable in production tracing of: 'occasional' state machines (node startup, snapshotting, reconfiguration), per-request profiling, active trace validation of ccfraft.tla; however that is all quite far away still.

For now I propose the following rough logging structure to make it easy to enable telemetry in production and simplify and improve the current state, primarily by adding log export to fluentd.

```mermaid
flowchart LR
subgraph Node["CCF node"]
PROD["Producer threads
LOG_*_FMT macros"]
TRACE["Producer threads
TRACE(, data)"]
CONS["Console sink
(stdout / journald)"]
FLU["Fluentd sink
per-thread queues
+ host reader thread"]
PROD --> CONS
PROD --> FLU
TRACE --> CONS
TRACE --> FLU
end
FLUENTD["fluentd
in_forward :24224"]
MDSD["mdsd"]
GENEVA["Geneva Logs"]
FLU -->|msgpack Forward
over TCP| FLUENTD
FLUENTD --> MDSD
MDSD --> GENEVA
```

## Configuration UX

This is what the config could roughly look like:
```
"logging": {
"sinks": [
{ "type": "console", "format": "Text", "level": "Info" },
{
"type": "fluentd_forward",
"level": "Debug",
"fluentd_tag": "ccf.node.1",
"address": "127.0.0.1:24224",
"thread_queue_capacity_bytes": "128KB",
"reconnect_backoff": "1s",
"connect_timeout": "2s",
}
]
}
```

The default should be to preserve compatibility with what we have currently.

Note: log level should still be controlled by the CLI but that should be the most verbose level that we allow and per-sink levels are clamped silently to that.
That allows us to have just FAIL to STDOUT but INFO or DEBUG to fluentd.

## Constraints
- Performance impact should be minimal
- Backpressure from fluentd cannot block the cchost process
- Likely only option is to drop packets, but log to stderr that it happened (w/ debounce and aggregate count)
- Log level captured at CLI
- How to log before fluentd connected? Just buffer?

## Rough implementation plan
- Micro-msgpack library for emitting the trace messages we need
- Export to fluentd
- Performance overhead test with trace logging of requests, for landmark style performance metrics

## Non-goals
- Logging semantics
- Message ordering vs timestamp
- Just sort the flow by timestamp
- Clock sync in a distributed deployment
- We're considering adding vector clocks or lamport clocks to the raft messages to allow a total order over these that respects causality.
- Other cases MUST NOT rely on cross instance timestamps
- Dynamic export sinks
- Private data in loglines controlled on a per-log-level basis, CLI arg takes care of this
- Migration of existing json logging to msgpack
- I think we want json as a midpoint which is machine and human readable
- TLS between cchost and fluentd
- There is support for TLS for fluentd `in_forward` but as its all on the same vnet (or localhost) this is unnecessary for v0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by locating the existing LOG_*_FMT and TRACE paths plus logging CLI and configuration handling. A complete implementation would preserve the current console behavior while adding configurable Fluentd export, non-blocking backpressure handling, CLI-capped levels, and performance coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.