linkedin / linkedin/ambry

BlobStore may not be respecting size limit in findEntriesSince()

Open
#1,048 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.8k
Forks
298
Avg merge
21h 43m
Merged PRs (30d)
9

Description

The heap dump of an OOM storage node revealed a log message from AmbryRequests.handleReplicaMetadataRequest() that was over 1GB.
The log line in question is:
publicAccessLogger.info("{} {} processingTime {}", replicaMetadataRequest, response, processingTime);
And the ReplicaMetadataResponse object apparently has a huge list.

We will add some logging/metrics to track when this happens and debug it.

java.lang.OutOfMemoryError: Java heap space
    at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:133)
    at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
    at java.io.Writer.write(Writer.java:157)
    at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:48)
    at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
    at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:369)
    at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
    at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
    at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
    at org.apache.log4j.Category.callAppenders(Category.java:206)
    at org.apache.log4j.Category.forcedLog(Category.java:391)
    at org.apache.log4j.Category.log(Category.java:856)
    at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:363)
    at com.github.ambry.server.AmbryRequests.handleReplicaMetadataRequest(AmbryRequests.java:647)
    at com.github.ambry.server.AmbryRequests.handleRequests(AmbryRequests.java:169)
    at com.github.ambry.server.RequestHandler.run(RequestHandler.java:46)
    at java.lang.Thread.run(Thread.java:745)```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing BlobStore.findEntriesSince() and AmbryRequests.handleReplicaMetadataRequest(), focusing on how the ReplicaMetadataResponse list can become unexpectedly large. Use the reported logging path and OOM trace to determine what observability is needed and what evidence would show that the size limit is being respected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.