apache / apache/rocketmq-clients
[Enhancement] Support injecting a custom logger for structured JSON logging
- Dominant language
- Java
- Stars
- 505
- Forks
- 313
- Avg merge
- 11h 28m
- Merged PRs (30d)
- 6
Description
### Before Creating the Enhancement Request
- [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.
### Programming Language of the Client
Go
### Summary
The Go client currently initializes a package-level logger during `init()` and writes logs to a rotating file under:
```text
$HOME/logs/rocketmq/rocketmq_client_go.log
```
It also does not support custom loggers or JSON format output.
### Motivation
The Go client currently initializes a package-level logger during `init()` and writes logs to a rotating file under:
```text
$HOME/logs/rocketmq/rocketmq_client_go.log
```
The output destination can be changed to stdout with:
```text
mq.consoleAppender.enabled=true
```
However, the main client logger always uses `zapcore.NewConsoleEncoder`, so its output remains plain text. There is currently no public API for replacing the package-level logger or configuring its encoder.
This causes several problems in containerized environments:
1. Containers commonly use a read-only root filesystem. When `$HOME` is `/root`, the client repeatedly reports errors such as:
```text
write error: can't make directories for new logfile:
mkdir /root/logs: read-only file system
```
2. Kubernetes applications generally write logs to stdout/stderr and delegate collection, rotation, and retention to the platform.
3. Applications using structured JSON logging cannot make RocketMQ client logs conform to the same schema.
4. `WithZapLogger` only configures the gRPC connection/interceptor logger. It does not replace the package-level `sugarBaseLogger` used by the main client implementation.
As a result, applications must either accept mixed log formats, mount a writable log directory, suppress RocketMQ logs, or maintain a patched fork.
## Proposed solution
Provide a supported API for injecting a custom `*zap.Logger` into the RocketMQ client.
### Describe the Solution You'd Like
### Option 1: Accept a custom Zap logger
Provide a public API for injecting a custom `*zap.Logger` into the RocketMQ client.
Example:
```go
logger := zap.New(
zapcore.NewCore(
zapcore.NewJSONEncoder(encoderConfig),
zapcore.AddSync(os.Stdout),
zap.InfoLevel,
),
)
rocketmq.SetLogger(logger)
```
The supplied logger should be used by the package-level client logger and inherited by newly created producers and consumers.
A minimal backward-compatible API could look like:
```go
// SetLogger replaces the package-level logger used by RocketMQ clients.
//
// It must be called before creating any producer or consumer.
func SetLogger(logger *zap.Logger) {
sugarBaseLogger = logger.Sugar()
}
```
The existing file-based logger and environment variables can remain the default, preserving current behavior for existing users.
If runtime replacement is not intended to be supported, the documentation should explicitly require `SetLogger` to be called before creating any client. This avoids synchronization and data-race concerns during normal operation.
A client-level option could also be provided:
```go
producer, err := rocketmq.NewProducer(
config,
rocketmq.WithTopics(topic),
rocketmq.WithLogger(logger),
)
```
A client-level option would avoid global state and allow different clients to use different loggers. Ideally, the supplied logger should also be propagated to the gRPC connection layer so users do not need to configure two separate logging systems.
#### Advantages
- Small implementation change.
- Preserves compatibility with the SDK's existing Zap implementation.
- Allows applications to configure JSON encoding, stdout output, log levels, sampling, and additional fields.
- Easier to introduce without refactoring every logging call site.
#### Disadvantages
- Exposes Zap as part of the public API.
- Applications using `slog`, Zerolog, Logrus, or an internal logger must create a Zap adapter.
- This will make it even more difficult to replace Zap within the SDK in the future.
### Option 2: Define a logger interface
Instead of accepting a concrete `*zap.Logger`, define a small logging interface. Applications could then integrate the RocketMQ client with their existing logging framework without depending on Zap-specific types.
For example:
```go
type Logger interface {
Debug(msg string, keyValues ...any)
Info(msg string, keyValues ...any)
Warn(msg string, keyValues ...any)
Error(msg string, keyValues ...any)
With(keyValues ...any) Logger
}
```
The logger could be supplied globally:
```go
rocketmq.SetLogger(newApplicationLogger())
```
or per client:
```go
producer, err := rocketmq.NewProducer(
config,
rocketmq.WithTopics(topic),
rocketmq.WithLogger(newApplicationLogger()),
)
```
The SDK could provide a default Zap-backed implementation to preserve the current behavior:
```go
type zapLogger struct {
logger *zap.SugaredLogger
}
func (l *zapLogger) Debug(msg string, keyValues ...any) {
l.logger.Debugw(msg, keyValues...)
}
func (l *zapLogger) Info(msg string, keyValues ...any) {
l.logger.Infow(msg, keyValues...)
}
func (l *zapLogger) Warn(msg string, keyValues ...any) {
l.logger.Warnw(msg, keyValues...)
}
func (l *zapLogger) Error(msg string, keyValues ...any) {
l.logger.Errorw(msg, keyValues...)
}
func (l *zapLogger) With(keyValues ...any) Logger {
return &zapLogger{
logger: l.logger.With(keyValues...),
}
}
```
Applications could implement the interface using `log/slog`:
```go
type applicationL
### Describe Alternatives You've Considered
### Enable the console appender
```text
mq.consoleAppender.enabled=true
```
This solves the read-only filesystem error by writing to stdout, but the main SDK logger still uses console encoding rather than JSON. Applications therefore produce mixed structured and unstructured logs.
### Change `rocketmq.client.logRoot`
Pointing the SDK at `/tmp` or another writable volume avoids the immediate error, but it still requires file lifecycle management inside the container.
It also conflicts with the common Kubernetes logging model, where applications write to stdout/stderr and the platform manages collection and rotation.
### Mount a writable volume at `$HOME/logs`
This works operationally but adds deployment complexity solely to satisfy an SDK logging implementation detail. It also requires separate collection and retention configuration for the RocketMQ log files.
### Use `WithZapLogger`
The existing `WithZapLogger` option configures the gRPC connection/interceptor logger only. It does not affect the package-level logger used by most RocketMQ client code, so it cannot provide consistent JSON output.
### Set the RocketMQ log level to `error`
Reducing the SDK log level limits the amount of unstructured output, but it does not solve the formatting issue and removes useful diagnostic information.
### Parse and re-encode console logs in the log collector
A log collector could parse the console-formatted SDK logs and wrap them in JSON. This is fragile because it depends on the SDK's text format and cannot reliably preserve
### Additional Context
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the package-level sugarBaseLogger, the existing WithZapLogger option, and the NewProducer client-creation path. Review how the main client and gRPC connection layer obtain their loggers, then define the supported injection scope and verify that newly created producers and consumers use the configured logger without breaking the current default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100