Avoid retries for FileNotFound exception in /segments controller API
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
For an incorrect segment download path, the `/segments` API failed with 500 http code - causing confusion as to whether the pinot controllers are broken. On debugging logs, this seems to be due to FileNotFound exception which got wrapped by AttemptsExceededException - delaying debugging process.
Proposed Improvements -
- Controller API - `/segments` should return a 404 error in case of such failures
- Retry should not be performed in case of FileNotFound exception
Error -
`Caused by: org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Exception while uploading segment: Operation failed after 3 attempts" while sending request: http://localhost:5436/segments to controller:`
Stacktrace for the API -
`
org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:74) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:124) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:132) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.downloadSegmentFileFromURI(PinotSegmentUploadDownloadRestletResource.java:461) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegment(PinotSegmentUploadDownloadRestletResource.java:277) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegmentAsJson(PinotSegmentUploadDownloadRestletResource.java:511)
`
Cause -
`
java.io.FileNotFoundException: /viewfs: at org.apache.hadoop.fs.viewfs.InodeTree.resolve(InodeTree.java:402) at org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:378) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2067) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2036) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2012) at com.uber.uPinot.filesystem.HDFSFileSystem.copyToLocalFile(HDFSFileSystem.java:335) at org.apache.pinot.spi.filesystem.NoClosePinotFS.copyToLocalFile(NoClosePinotFS.java:98) at org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher.fetchSegmentToLocalWithoutRetry(PinotFSSegmentFetcher.java:31) at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.lambda$fetchSegmentToLocal$0(BaseSegmentFetcher.java:76) at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:74) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:124) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:132) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.downloadSegmentFileFromURI(PinotSegmentUploadDownloadRestletResource.java:461) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegment(PinotSegmentUploadDownloadRestletResource.java:277) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegmentAsJson(PinotSegmentUploadDownloadRestletResource.java:511)
`
Contributor guide
Research direction
Start by reading BaseSegmentFetcher.java and PinotFSSegmentFetcher.java around the stack-trace locations, then trace the /segments flow in PinotSegmentUploadDownloadRestletResource.java. Verify the behavior for a missing segment path: the API should return 404 and the FileNotFoundException should not trigger retries or become an AttemptsExceededException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100