Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: DFM25B81QM9RZ2WX, Extended Request ID: 7vSTw22PgYLApWtVYc4blcMDFPRq1NOUG/5tKNqvH/YWCQwyhEO87rv6OOL0AFRf8Ia/+M4voOtLHW656U10Ng==)

I am unable to start my pinot controller pod in AWS EKS environment.

**For that I have added configurations in values.template.yaml for pinot to connect it from aws s3**

As I have confirmed my serviceaccount have correct permissions and policies to access AWS S3

As I have tested it while exec into another pod using the same service account:

```
in the pinot pod
apt-get update
apt-get install awscli
# aws s3 ls s3://perceptdata-pinot
PRE controller-data/
PRE pinot-data/
```

**Configurations:**

```
# Extra configs will be appended to pinot-controller.conf file
extra:
configs: |-
pinot.set.instance.id.to.hostname=true
controller.task.scheduler.enabled=true
controller.disable.ingestion.groovy=false
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=eu-west-1
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

```
**The data directory for aws s3 bucket is:**

```
data:
dir: s3://perceptdata-pinot/controller-data
```

The following Error logs I am facing:

```
Initializing ControllerFilePathProvider
Data directory: s3://perceptdata-pinot/controller-data/
Failed to start a Pinot [CONTROLLER] at 17.271 since launch
java.lang.RuntimeException: Caught exception while initializing ControllerFilePathProvider
at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:569) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:380) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:328) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:118) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:87) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:251) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:250) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:196) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:187) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
Caused by: org.apache.pinot.controller.api.resources.InvalidControllerConfigException: Caught exception while initializing file upload path provider
at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.(ControllerFilePathProvider.java:107) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
... 20 more
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: TMMVB33627KABQ7N, Extended Request ID: VOWG6S7PM21PAQ6FNeGPIMFeMFiz/CnWf+MoJ7pyGTktnYuOEV7mzjNGnXe8DXx4ZP1jCDqXnco=)
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$5(BaseClientHandler.java:229) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:5614) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.plugin.filesystem.S3PinotFS.isDirectory(S3PinotFS.java:572) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.plugin.filesystem.S3PinotFS.exists(S3PinotFS.java:437) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.spi.filesystem.NoClosePinotFS.exists(NoClosePinotFS.java:74) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.(ControllerFilePathProvider.java:71) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
... 20 more
Shutting down Pinot Service Manager with all running Pinot instances...
Shutting down Pinot Service Manager admin application...
Deregistering service status handler

```

**As I am using a service account to access aws in my pinot-controller pod**
as the specs of service account are as:

```
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::333870179331:role/pinot
meta.helm.sh/release-name: pinot
meta.helm.sh/release-namespace: pinot-new
creationTimestamp: "2023-08-25T06:54:37Z"
labels:
app: pinot
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/version: 0.2.7
helm.sh/chart: pinot-0.2.7
heritage: Helm
release: pinot
name: pinot
namespace: pinot-new
resourceVersion: "389804078"
uid: 488a7bf2-15fe-4aac-a3ad-e82c5515b3c9
```

This is the policy for pinot :

```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PinotAccessPolicy",
"Effect": "Allow",
"Action": [
"*"
],
"Resource": [
"arn:aws:s3:::perceptdata-pinot/*",
"arn:aws:s3:::perceptdata-pinot"
]
}
]
}
```

Start with values.template.yaml and the startup path in BaseControllerStarter.java and ControllerFilePathProvider.java, then inspect the S3PinotFS configuration used for the s3:// data directory. Reproduce the controller startup in AWS EKS with the shown service account and configuration, comparing it with the successful aws s3 ls check. Done means the Pinot controller starts without the S3 Access Denied error.

Written by the indexing model from the issue text.