GET requests sometimes result in 500 Internal error responses
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.8k
- Forks
- 298
- Avg merge
- 21h 43m
- Merged PRs (30d)
- 9
Description
I use an Ambry instance for saving and serving media files (images and videos), and start having problem with some requests failing with status: 500 Internal Server Error and following exception trace in the log:
[2017-10-10 16:02:52,366] ERROR Internal error handling request /AAEAAQAAAAAAAAAAAAAAJGExMDlhYWFjLWNjNGYtNDVjOS04ZjliLTUxMDY0MTdhZWY5OQ with method GET (com.github.ambry.rest.NettyResponseChannel)
com.github.ambry.rest.RestServiceException: com.github.ambry.router.RouterException: Timed out waiting for a response Error: OperationTimedOut
at com.github.ambry.frontend.AmbryBlobStorageService.submitResponse(AmbryBlobStorageService.java:322)
at com.github.ambry.frontend.AmbryBlobStorageService$GetCallback.onCompletion(AmbryBlobStorageService.java:786)
at com.github.ambry.frontend.AmbryBlobStorageService$GetCallback.onCompletion(AmbryBlobStorageService.java:698)
at com.github.ambry.router.NonBlockingRouter$1.onCompletion(NonBlockingRouter.java:155)
at com.github.ambry.router.NonBlockingRouter$1.onCompletion(NonBlockingRouter.java:149)
at com.github.ambry.router.NonBlockingRouter.completeOperation(NonBlockingRouter.java:392)
at com.github.ambry.router.GetBlobOperation.abort(GetBlobOperation.java:146)
at com.github.ambry.router.GetBlobOperation.poll(GetBlobOperation.java:263)
at com.github.ambry.router.GetManager.poll(GetManager.java:155)
at com.github.ambry.router.NonBlockingRouter$OperationController.pollForRequests(NonBlockingRouter.java:522)
at com.github.ambry.router.NonBlockingRouter$OperationController.run(NonBlockingRouter.java:575)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.github.ambry.router.RouterException: Timed out waiting for a response Error: OperationTimedOut
at com.github.ambry.router.GetBlobOperation$GetChunk.cleanupExpiredInFlightRequests(GetBlobOperation.java:553)
at com.github.ambry.router.GetBlobOperation$GetChunk.poll(GetBlobOperation.java:530)
at com.github.ambry.router.GetBlobOperation.poll(GetBlobOperation.java:232)
... 4 more
- Ambry version: recent built from commit e1a688eccb040934c3e6e94281da6c2b9826522d
- JVM version: 1.8.0_131
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the stack-trace entry points in AmbryBlobStorageService.java, NonBlockingRouter.java, and GetBlobOperation.java, focusing on GetChunk cleanupExpiredInFlightRequests and the OperationTimedOut path. Reproduce or trace the failing GET and determine the cause of the timeout and the expected handling of the response; done should include a verified resolution for the reported 500 errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100