BlobStore may not be respecting size limit in findEntriesSince()
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.8k
- Forks
- 298
- Avg merge
- 21h 43m
- Merged PRs (30d)
- 9
Description
The heap dump of an OOM storage node revealed a log message from AmbryRequests.handleReplicaMetadataRequest() that was over 1GB.
The log line in question is:
publicAccessLogger.info("{} {} processingTime {}", replicaMetadataRequest, response, processingTime);
And the ReplicaMetadataResponse object apparently has a huge list.
We will add some logging/metrics to track when this happens and debug it.
java.lang.OutOfMemoryError: Java heap space
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:133)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
at java.io.Writer.write(Writer.java:157)
at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:48)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:369)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:363)
at com.github.ambry.server.AmbryRequests.handleReplicaMetadataRequest(AmbryRequests.java:647)
at com.github.ambry.server.AmbryRequests.handleRequests(AmbryRequests.java:169)
at com.github.ambry.server.RequestHandler.run(RequestHandler.java:46)
at java.lang.Thread.run(Thread.java:745)```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing BlobStore.findEntriesSince() and AmbryRequests.handleReplicaMetadataRequest(), focusing on how the ReplicaMetadataResponse list can become unexpectedly large. Use the reported logging path and OOM trace to determine what observability is needed and what evidence would show that the size limit is being respected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100