linkedin / linkedin/ambry

GET requests sometimes result in 500 Internal error responses

Open
#783 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.8k
Forks
298
Avg merge
21h 43m
Merged PRs (30d)
9

Description

I use an Ambry instance for saving and serving media files (images and videos), and start having problem with some requests failing with status: 500 Internal Server Error and following exception trace in the log:

    [2017-10-10 16:02:52,366] ERROR Internal error handling request /AAEAAQAAAAAAAAAAAAAAJGExMDlhYWFjLWNjNGYtNDVjOS04ZjliLTUxMDY0MTdhZWY5OQ with method GET (com.github.ambry.rest.NettyResponseChannel)
    com.github.ambry.rest.RestServiceException: com.github.ambry.router.RouterException: Timed out waiting for a response Error: OperationTimedOut
            at com.github.ambry.frontend.AmbryBlobStorageService.submitResponse(AmbryBlobStorageService.java:322)
            at com.github.ambry.frontend.AmbryBlobStorageService$GetCallback.onCompletion(AmbryBlobStorageService.java:786)
            at com.github.ambry.frontend.AmbryBlobStorageService$GetCallback.onCompletion(AmbryBlobStorageService.java:698)
            at com.github.ambry.router.NonBlockingRouter$1.onCompletion(NonBlockingRouter.java:155)
            at com.github.ambry.router.NonBlockingRouter$1.onCompletion(NonBlockingRouter.java:149)
            at com.github.ambry.router.NonBlockingRouter.completeOperation(NonBlockingRouter.java:392)
            at com.github.ambry.router.GetBlobOperation.abort(GetBlobOperation.java:146)
            at com.github.ambry.router.GetBlobOperation.poll(GetBlobOperation.java:263)
            at com.github.ambry.router.GetManager.poll(GetManager.java:155)
            at com.github.ambry.router.NonBlockingRouter$OperationController.pollForRequests(NonBlockingRouter.java:522)
            at com.github.ambry.router.NonBlockingRouter$OperationController.run(NonBlockingRouter.java:575)
            at java.lang.Thread.run(Thread.java:748)
    Caused by: com.github.ambry.router.RouterException: Timed out waiting for a response Error: OperationTimedOut
            at com.github.ambry.router.GetBlobOperation$GetChunk.cleanupExpiredInFlightRequests(GetBlobOperation.java:553)
            at com.github.ambry.router.GetBlobOperation$GetChunk.poll(GetBlobOperation.java:530)
            at com.github.ambry.router.GetBlobOperation.poll(GetBlobOperation.java:232)
            ... 4 more
  • Ambry version: recent built from commit e1a688eccb040934c3e6e94281da6c2b9826522d
  • JVM version: 1.8.0_131

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the stack-trace entry points in AmbryBlobStorageService.java, NonBlockingRouter.java, and GetBlobOperation.java, focusing on GetChunk cleanupExpiredInFlightRequests and the OperationTimedOut path. Reproduce or trace the failing GET and determine the cause of the timeout and the expected handling of the response; done should include a verified resolution for the reported 500 errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.