Azure / Azure/azure-sdk-for-java
[cosmos] ChangeFeedProcessor: CosmosException logging can produce multi-MB log lines due to full diagnostics serialization
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 2.2k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 178
Description
**Component**: com.azure.cosmos.implementation.changefeed.epkversion.PartitionProcessorImpl (and pkversion equivalent)
**Summary**
When a CosmosException is logged at WARN level in the ChangeFeedProcessor's partition processing loop, the logger invokes CosmosException.getMessage() to render the exception. This method serializes the full CosmosDiagnostics payload as JSON — including request timelines, region contacts, retry history, and metadata. Under heavy db connection load (i.e. thousands of connections) this can result in multi-MB log lines.
**Where it occurs**
PartitionProcessorImpl.run() → onErrorResume handler (the error classification/recovery path in the per-partition change feed polling loop).
This path is triggered on every CosmosException from createDocumentChangeFeedQuery, including transient errors (429 throttles, timeouts, connectivity issues) which are expected and frequent under load. Since each leased partition runs its own processor instance, a busy ChangeFeedProcessor can hit this path concurrently across many partitions.
**Root cause**
CosmosException.getMessage() (line 263 of CosmosException.java) builds a JSON string containing the full CosmosDiagnostics:
```java
public String getMessage() {
ObjectNode messageNode = mapper.createObjectNode();
messageNode.put("innerErrorMessage", innerErrorMessage());
if (cosmosDiagnostics != null) {
cosmosDiagnostics.fillCosmosDiagnostics(messageNode, null);
}
return mapper.writeValueAsString(messageNode);
}
```
The json message string contains the RntdbTransportClient and RntbdEndpoint objects in the following structure:
```json
{
"id": ,
"isClosed": ,
"configuration": ,
"serviceEndpoints": {
"count": ,
"items": [ ]
}
}
```
When the exception is passed to logger.warn(message, exception), SLF4J calls toString() → getMessage(), producing the potentially massive payload.
**Impact**
- massive individual log lines (we've seen them exceeding 30+ MB)
- Risk of log ingestion pipeline throttling or rejection
- Increased log storage costs
- Potential memory pressure from repeated serialization of diagnostics on transient errors in a hot loop
**PossibleFix**
Split existing log into two logging tiers:
- WARN level: Log with CosmosException.getShortMessage() (lightweight, no diagnostics)
- DEBUG level: Log the full exception (with diagnostics + stack trace) for deep troubleshooting
This preserves the log message signature for pattern matching while eliminating the excessive payload at default log levels.
Contributor guide
Assessment
This issue has not been assessed yet.