Access denied error when using Task IAM Role Credentials

I'm trying to get a job to run on AWS Batch. The job runs in a docker container, using credentials generated for a Task IAM Role.

I can verify that the role has permissions to the bucket, because I can access the exact same files if I run an aws s3 cp command (as shown in the example below).

However, when I try to access the files through s3fs, I get access denied errors. It seems to fail on this line, where it tries to list the contents of the file after failing to find the object via a head_object call.

I suspect there may be a bug in s3fs, or in the particular combination of boto, http, and s3 libraries.

Here is a minimal reproducible example:

Shell script for the job:

#!/bin/bash

AWS_CREDENTIALS=$(curl http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)

export AWS_DEFAULT_REGION=us-east-1

export AWS_ACCESS_KEY_ID=$(echo "$AWS_CREDENTIALS" | jq .AccessKeyId -r)

export AWS_SECRET_ACCESS_KEY=$(echo "$AWS_CREDENTIALS" | jq .SecretAccessKey -r)

export AWS_SESSION_TOKEN=$(echo "$AWS_CREDENTIALS" | jq .Token -r)

echo "AWS_ACCESS_KEY_ID=<$AWS_ACCESS_KEY_ID>"

echo "AWS_SECRET_ACCESS_KEY=<$(cat <(echo "$AWS_SECRET_ACCESS_KEY" | head -c 6) <(echo -n "...") <(echo "$AWS_SECRET_ACCESS_KEY" | tail -c 6))>"

echo "AWS_SESSION_TOKEN=<$(cat <(echo "$AWS_SESSION_TOKEN" | head -c 6) <(echo -n "...") <(echo "$AWS_SESSION_TOKEN" | tail -c 6))>"

# Succeeds!
aws s3 ls s3://company-dvc/repo/

# Succeeds!
aws s3 cp s3://company-dvc/repo/00/0e4343c163bd70df0a6f9d81e1b4d2 mycopy.txt

# Fails!
python3 download_via_s3fs.py

download_via_s3fs.py:

import os

import s3fs

# Just to make sure we're reading the credentials correctly.
print(os.environ["AWS_ACCESS_KEY_ID"])
print(os.environ["AWS_SECRET_ACCESS_KEY"])
print(os.environ["AWS_SESSION_TOKEN"])

print("running with credentials")
fs = s3fs.S3FileSystem(
    key=os.environ["AWS_ACCESS_KEY_ID"],
    secret=os.environ["AWS_SECRET_ACCESS_KEY"],
    token=os.environ["AWS_SESSION_TOKEN"],
    client_kwargs={"region_name": "us-east-1"}
)

# Fails with "access denied" on ListObjectV2
print(fs.exists("company-dvc/repo/00/0e4343c163bd70df0a6f9d81e1b4d2"))

Terraform for IAM role:

data "aws_iam_policy_document" "standard-batch-job-role" {
  # S3 read access to related buckets
  statement {
    actions = [
      "s3:Get*",
      "s3:List*",
    ]
    resources = [
      data.aws_s3_bucket.company-dvc.arn,
      "${data.aws_s3_bucket.company-dvc.arn}/*",
    ]
    effect = "Allow"
  }
}

Environment

OS: Ubuntu 20.04
Python: 3.10
s3fs: 2023.1.0
boto3: 1.24.59

Reproduce the issue with the provided shell script and download_via_s3fs.py, then inspect s3fs/core.py around line 1264 and the fs.exists path. Compare the successful AWS CLI calls with s3fs requests using the task-role credentials. Done means the access-denied behavior is explained and, if a defect is confirmed, covered by a regression test.