apache / apache/pinot

Avoid retries for FileNotFound exception in /segments controller API

Open
#16,231 3 comments 0 reactions 0 assignees View on GitHub
bug rest-api
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

For an incorrect segment download path, the `/segments` API failed with 500 http code - causing confusion as to whether the pinot controllers are broken. On debugging logs, this seems to be due to FileNotFound exception which got wrapped by AttemptsExceededException - delaying debugging process.

Proposed Improvements -

- Controller API - `/segments` should return a 404 error in case of such failures
- Retry should not be performed in case of FileNotFound exception

Error -
`Caused by: org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Exception while uploading segment: Operation failed after 3 attempts" while sending request: http://localhost:5436/segments to controller:`

Stacktrace for the API -
`
org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:74) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:124) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:132) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.downloadSegmentFileFromURI(PinotSegmentUploadDownloadRestletResource.java:461) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegment(PinotSegmentUploadDownloadRestletResource.java:277) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegmentAsJson(PinotSegmentUploadDownloadRestletResource.java:511)
`

Cause -
`
java.io.FileNotFoundException: /viewfs: at org.apache.hadoop.fs.viewfs.InodeTree.resolve(InodeTree.java:402) at org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:378) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2067) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2036) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2012) at com.uber.uPinot.filesystem.HDFSFileSystem.copyToLocalFile(HDFSFileSystem.java:335) at org.apache.pinot.spi.filesystem.NoClosePinotFS.copyToLocalFile(NoClosePinotFS.java:98) at org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher.fetchSegmentToLocalWithoutRetry(PinotFSSegmentFetcher.java:31) at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.lambda$fetchSegmentToLocal$0(BaseSegmentFetcher.java:76) at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:74) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:124) at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:132) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.downloadSegmentFileFromURI(PinotSegmentUploadDownloadRestletResource.java:461) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegment(PinotSegmentUploadDownloadRestletResource.java:277) at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegmentAsJson(PinotSegmentUploadDownloadRestletResource.java:511)
`

Contributor guide

Open the contributing guide

Research direction

Start by reading BaseSegmentFetcher.java and PinotFSSegmentFetcher.java around the stack-trace locations, then trace the /segments flow in PinotSegmentUploadDownloadRestletResource.java. Verify the behavior for a missing segment path: the API should return 404 and the FileNotFoundException should not trigger retries or become an AttemptsExceededException.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.