Introduce rate limited or sampled logging
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
### What problem are you trying to solve?
## Background
While investigating logs emitted by `linkerd-proxy`, I noticed that many of the same logs get emitted without adding extra information. For example, the following (DEBUG) logs contain a similar message:
```
[ 1239.102811s] DEBUG ThreadId(02) linkerd_app_inbound::policy::http: Request authorized server.group= server.kind=default server.name=all-unauthenticated route.group= route.kind=default route.name=probe authz.group= authz.kind=default authz.name=probe client.tls=None(NoClientHello) client.ip=10.244.0.1
[ 1241.924744s] DEBUG ThreadId(02) linkerd_app_inbound::policy::http: Request authorized server.group= server.kind=default server.name=all-unauthenticated route.group= route.kind=default route.name=probe authz.group= authz.kind=default authz.name=probe client.tls=None(NoClientHello) client.ip=10.244.0.1
[ 1248.315745s] DEBUG ThreadId(02) linkerd_app_inbound::policy::http: Request authorized server.group= server.kind=default server.name=all-unauthenticated route.group= route.kind=default route.name=default authz.group= authz.kind=default authz.name=all-unauthenticated client.tls=Some(Established { client_id: Some(ClientId(Dns(Name("prometheus.linkerd-viz.serviceaccount.identity.linkerd.cluster.local")))), negotiated_protocol: None }) client.ip=10.244.0.14
[ 1249.103011s] DEBUG ThreadId(02) linkerd_app_inbound::policy::http: Request authorized server.group= server.kind=default server.name=all-unauthenticated route.group= route.kind=default route.name=probe authz.group= authz.kind=default authz.name=probe client.tls=None(NoClientHello) client.ip=10.244.0.1
[ 1251.925220s] DEBUG ThreadId(02) linkerd_app_inbound::policy::http: Request authorized server.group= server.kind=default server.name=all-unauthenticated route.group= route.kind=default route.name=probe authz.group= authz.kind=default authz.name=probe client.tls=None(NoClientHello) client.ip=10.244.0.1
[ 1258.316034s] DEBUG ThreadId(02) linkerd_app_inbound::policy::http: Request authorized server.group= server.kind=default server.name=all-unauthenticated route.group= route.kind=default route.name=default authz.group= authz.kind=default authz.name=all-unauthenticated client.tls=Some(Established { client_id: Some(ClientId(Dns(Name("prometheus.linkerd-viz.serviceaccount.identity.linkerd.cluster.local")))), negotiated_protocol: None }) client.ip=10.244.0.14
[ 1259.102343s] DEBUG ThreadId(02) linkerd_app_inbound::policy::http: Request authorized server.group= server.kind=default server.name=all-unauthenticated route.group= route.kind=default route.name=probe authz.group= authz.kind=default authz.name=probe client.tls=None(NoClientHello) client.ip=10.244.0.1
```
## Issues
1. Even though `DEBUG` logs are normally noisy, I believe other log levels could suffer from the same potential issue. In this case, logs are being emitted when they don't necessarily need to be.
1. Some linkerd users may operate in environments where writing every event to a log is expensive. Especially when the service mesh and L7 applications are performing as expected. I am thinking of workloads with high volumes of traffic.
1. If the service mesh is operating normally, then the extra IO spent logging is more of a performance hit.
### How should the problem be solved?
I think linkerd could offer different logging modes that are similar to the log levels that users can configure. The "mode" would indicate whether verbose logging is turned on regardless of log level, or sampled/rate limited logging is enabled. The current default is verbose logging but if sampled or rate limited logging is selected, then linkerd could emit messages with aggregated stats.
I was thinking of something like:
```
[ 1239.102811s] DEBUG ThreadId(02) (occurred N times) linkerd_app_inbound::policy::http: Request authorized server.group= server.kind=default server.name=all-unauthenticated route.group= route.kind=default route.name=probe authz.group= authz.kind=default authz.name=probe client.tls=None(NoClientHello) client.ip=10.244.0.1
[ 1248.315745s] DEBUG ThreadId(02) (occurred N times) linkerd_app_inbound::policy::http: Request authorized server.group= server.kind=default server.name=all-unauthenticated route.group= route.kind=default route.name=default authz.group= authz.kind=default authz.name=all-unauthenticated client.tls=Some(Established { client_id: Some(ClientId(Dns(Name("prometheus.linkerd-viz.serviceaccount.identity.linkerd.cluster.local")))), negotiated_protocol: None }) client.ip=10.244.0.14
```
If the messages were sampled, then I don't think the aggregated stat would be worthwhile.
The default "verbose" mode is still useful and could be switched on during an incident where users need more visibility than what HTTP Access Logging or Tap provides.
### Any alternatives you've considered?
TBD.
I did propose a rate limited variant but that might only ever be useful if many events happen around the same time. The sampled variant is a little bit simpler but would skew logs to the most frequent kinds of events that happen.
### How would users interact with this feature?
Similar to the HTTP Access Logging, I think users could add an annotation to the namespace or workload that indicates the desired logging "mode". E.g.,
```
config.linkerd.io/logging-mode: verbose|sampled|rate-limited
```
I'm open to suggestions on the naming and approach.
### Would you like to work on this feature?
yes
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with linkerd-proxy's existing logging behavior and the proposed config.linkerd.io/logging-mode annotation; the issue names no files or tests. Clarify whether verbose, sampled, or rate-limited behavior is in scope, define the configuration and aggregation semantics, and consider the work done when a selected mode produces the expected log behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100