Logging errors in events package

Use case

The events package is used by containerd [both client and server] to build the mechanism for pub/sub for container events.

What is the issue?

The issue containerd has hit an issue where certain events can contain fields that users do not want to be logged. e.g. containerd's TaskExecCreate event can contain an I/O field which when containers are configured with shim loggers can contain environment variables for required credentials to persist container logs to log services.

The challenge is with the current design users can only influence events package logs by configuring the log level for the process which consumes it. For the above use case, this means disabling containerd debug logging which isn't really helpful.

Solutions considered

New queue function for returning asynchronous write errors via a channel

A read-only error channel will be a queue field for returning async write errors for users to handle based on their use case.

func NewQueue(dst Sink) (*Queue, <-chan error) {
    ...
}

Trade-offs

The downside of this approach users invoking Queue.Write(event) always receive nil error return. Users must be aware that write errors are returned via the error channel when the queue was created. The benefit of this approach is it maintains backwards compatibility with existing interfaces.

Add context.Context support to events package

The events package was created before context.Context was mainstream in Go developers' toolboxes. This package would benefit from an updated implementation with asynchronous Sink implementations (i.e. queue) taking a context.

Trade-offs

The downside of this approach is it may requires interface changes for at least Broadcaster and Queue implementations. The benefits is all logging can be updated to use github.com/containerd/log package for context logging give users more control on how library logs are handled.

Reduce logging level to trace

Name says it all

Trade-offs

Short-term solution, doesn't really solve the painpoint.

Remove all logs, dropped events silently ignored

Name says it all

Trade-offs

Not really user friendly. These logs are probably useful to someone (right?)

Start by reviewing the events package interfaces and the Queue.Write and NewQueue entry points described in the issue, along with Broadcaster and Sink implementations. Compare the proposed error-channel and context.Context approaches, then identify the compatibility and logging behavior that must be settled. Done means an agreed implementation direction with corresponding API and error-handling changes.