[source-s3] Can not create source for s3 constantly fails with any options
- Dominant language
- Python
- Stars
- 22.1k
- Forks
- 5.3k
- PR merge metrics
- PR metrics pending
Description
### Connector Name
S3
### Connector Version
4.14.1
### What step the error happened?
Configuring a new connector
### Relevant information
abctl version:v0.30.1
Installed on my local Fedora: abctl local install
runs in: docker + local k8s on top of docker
Installed Helm Chart airbyte/airbyte:
Name: airbyte-abctl
Namespace: airbyte-abctl
Version: 2.0.10
AppVersion: 1.8.1
Release: 1
In aws my user has this policy attached, from any clien as aws cli, spark, cyberduck etc. it works:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::my_bucket",
"arn:aws:s3:::my_bucket/*"
],
"Effect": "Allow"
}
]
}
```
Trying to configure S3 connector:
Source name: S3
Bucket: my_bucket_name/path/to/my/parquet/files/
Optional fields:
AWS Access Key ID: my_key
AWS Secret Access Key: my_secret
Endpoint: s3.us-east-1.amazonaws.com
Delivery Method: Replicate Records
or
Delivery Method: Copy Raw Files
Test the source (UI):
```
Configuration check failed
'No streams are available for source SourceS3. This is probably an issue with the connector. Please verify that your configuration provides permissions to list and read files from the source. Contact support if you are unable to resolve this issue.'
```
Logs (in k8s pod):
```
2025-08-28 12:18:11 INFO 2025-08-28 12:18:11 info
2025-08-28 12:18:11 INFO 2025-08-28 12:18:11 info ----- START CHECK -----
2025-08-28 12:18:11 INFO 2025-08-28 12:18:11 info
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Connector exited, processing output
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Output file jobOutput.json found
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Connector exited with exit code 0
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Reading messages from protocol version 0.2.0
2025-08-28 12:18:16 ERROR 2025-08-28 12:18:16 error Check failed
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Checking for optional control message...
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Optional control message not found. Skipping...
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Writing output of 69589781-7828-43c5-9f63-8925b1c1ccc2_dc9c84f4-e17c-423e-a59e-4da45e69f516_0_check to the doc store
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info Marking workload 69589781-7828-43c5-9f63-8925b1c1ccc2_dc9c84f4-e17c-423e-a59e-4da45e69f516_0_check as successful
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info Deliberately exiting process with code 0.
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info ----- END CHECK -----
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info
```
Trying to configure S3 connector:
Source name: S3
Bucket: my_bucket_name
**The list of streams to sync**
Format: Parquet Format
Name: my_stream
Optional fields
Globs: **
Days To Sync If History Is Full: 300 (set different values here 3, 10, 100)
Schemaless: off
Validation Policy: Emit Record
Optional fields:
AWS Access Key ID: my_key
AWS Secret Access Key: my_secret
Endpoint: s3.us-east-1.amazonaws.com
Delivery Method: Replicate Records
or
Delivery Method: Copy Raw Files
Test the source (UI):
```
Configuration check failed
Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 100, in _check_list_files file = next(iter(stream.get_files())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 140, in get_matching_files s3 = self.s3_client ^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 80, in s3_client self._s3_client = boto3.client( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/boto3/__init__.py", line 92, in client return _get_default_session().client(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/boto3/session.py", line 322, in client return self._session.create_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/context.py", line 123, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 1019, in create_client client = client_creator.create_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 166, in create_client client_args = self._get_client_args( ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 525, in _get_client_args return args_creator.get_client_args( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/args.py", line 163, in get_client_args endpoint = endpoint_creator.create_endpoint( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 408, in create_endpoint raise ValueError(f"Invalid endpoint: {endpoint_url}") ValueError: Invalid endpoint: s3.us-east-1.amazonaws.com The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 79, in check_availability_and_parsability file = self._check_list_files(stream) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 106, in _check_list_files raise CheckAvailabilityError( airbyte_cdk.sources.file_based.exceptions.CheckAvailabilityError: Error listing files. Please check the credentials provided in the config and verify that they provide permission to list files. Contact Support if you need assistance. stream=my_stream
```
Logs (k8s pod):
```
2025-08-28 12:33:40 INFO 2025-08-28 12:33:40 info
2025-08-28 12:33:40 INFO 2025-08-28 12:33:40 info ----- START CHECK -----
2025-08-28 12:33:40 INFO 2025-08-28 12:33:40 info
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Connector exited, processing output
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Output file jobOutput.json found
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Connector exited with exit code 0
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Reading messages from protocol version 0.2.0
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Checking for optional control message...
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Optional control message not found. Skipping...
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Writing output of 69589781-7828-43c5-9f63-8925b1c1ccc2_ba4e7de7-3170-4824-b79b-cb6d0152363c_0_check to the doc store
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Marking workload 69589781-7828-43c5-9f63-8925b1c1ccc2_ba4e7de7-3170-4824-b79b-cb6d0152363c_0_check as successful
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Deliberately exiting process with code 0.
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info ----- END CHECK -----
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info
```
### Relevant log output
```shell
Test the source (UI):
Configuration check failed
'No streams are available for source SourceS3. This is probably an issue with the connector. Please verify that your configuration provides permissions to list and read files from the source. Contact support if you are unable to resolve this issue.'
Logs (in k8s pod):
2025-08-28 12:18:11 INFO 2025-08-28 12:18:11 info
2025-08-28 12:18:11 INFO 2025-08-28 12:18:11 info ----- START CHECK -----
2025-08-28 12:18:11 INFO 2025-08-28 12:18:11 info
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Connector exited, processing output
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Output file jobOutput.json found
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Connector exited with exit code 0
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Reading messages from protocol version 0.2.0
2025-08-28 12:18:16 ERROR 2025-08-28 12:18:16 error Check failed
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Checking for optional control message...
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Optional control message not found. Skipping...
2025-08-28 12:18:16 INFO 2025-08-28 12:18:16 info Writing output of 69589781-7828-43c5-9f63-8925b1c1ccc2_dc9c84f4-e17c-423e-a59e-4da45e69f516_0_check to the doc store
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info Marking workload 69589781-7828-43c5-9f63-8925b1c1ccc2_dc9c84f4-e17c-423e-a59e-4da45e69f516_0_check as successful
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info Deliberately exiting process with code 0.
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info ----- END CHECK -----
2025-08-28 12:18:17 INFO 2025-08-28 12:18:17 info
Test the source (UI):
Configuration check failed
Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 100, in _check_list_files file = next(iter(stream.get_files())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 140, in get_matching_files s3 = self.s3_client ^^^^^^^^^^^^^^ File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 80, in s3_client self._s3_client = boto3.client( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/boto3/__init__.py", line 92, in client return _get_default_session().client(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/boto3/session.py", line 322, in client return self._session.create_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/context.py", line 123, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/session.py", line 1019, in create_client client = client_creator.create_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 166, in create_client client_args = self._get_client_args( ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 525, in _get_client_args return args_creator.get_client_args( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/args.py", line 163, in get_client_args endpoint = endpoint_creator.create_endpoint( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/endpoint.py", line 408, in create_endpoint raise ValueError(f"Invalid endpoint: {endpoint_url}") ValueError: Invalid endpoint: s3.us-east-1.amazonaws.com The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 79, in check_availability_and_parsability file = self._check_list_files(stream) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 106, in _check_list_files raise CheckAvailabilityError( airbyte_cdk.sources.file_based.exceptions.CheckAvailabilityError: Error listing files. Please check the credentials provided in the config and verify that they provide permission to list files. Contact Support if you need assistance. stream=my_stream
Logs (k8s pod):
2025-08-28 12:33:40 INFO 2025-08-28 12:33:40 info
2025-08-28 12:33:40 INFO 2025-08-28 12:33:40 info ----- START CHECK -----
2025-08-28 12:33:40 INFO 2025-08-28 12:33:40 info
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Connector exited, processing output
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Output file jobOutput.json found
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Connector exited with exit code 0
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Reading messages from protocol version 0.2.0
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Checking for optional control message...
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Optional control message not found. Skipping...
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Writing output of 69589781-7828-43c5-9f63-8925b1c1ccc2_ba4e7de7-3170-4824-b79b-cb6d0152363c_0_check to the doc store
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Marking workload 69589781-7828-43c5-9f63-8925b1c1ccc2_ba4e7de7-3170-4824-b79b-cb6d0152363c_0_check as successful
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info Deliberately exiting process with code 0.
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info ----- END CHECK -----
2025-08-28 12:33:45 INFO 2025-08-28 12:33:45 info
```
### Contribute
- [ ] Yes, I want to contribute
Contributor guide
Assessment
This issue has not been assessed yet.