S3 destination giving botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### dlt version
1.13.0
### Describe the problem
I'm trying to run this simple code using a S3 bucket as destination. I'm using environment variables to put the config parameters.
```
import dlt
from dotenv import load_dotenv
load_dotenv()
def hello_world_resource():
@dlt.resource(name="hello_world", write_disposition="replace", file_format="parquet")
def resource():
data = [
{"message": "Hello World", "idioma": "english", "id": 1},
]
for item in data:
yield item
return resource()
pipeline = dlt.pipeline(
pipeline_name="hello_world_pipeline",
destination="filesystem",
dataset_name="hello_world_dataset",
)
if __name__ == "__main__":
resource = hello_world_resource()
info = pipeline.run(resource)
```
When I use `BUCKET_URL=somefolder`, this code works fine.
When I use `BUCKET_URL=s3://mybucket-that-exists` I'm getting the error below. My .env file is configured as follows:
```
AWS_ACCESS_KEY_ID="secret"
AWS_SECRET_ACCESS_KEY="secret"
AWS_REGION="us-east-1"
BUCKET_URL='s3://mybucket-that-exists'
```
Error:
```
Traceback (most recent call last):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 114, in _error_wrapper
return await func(*args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\aiobotocore\client.py", line 412, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 783, in _sync_destination
remote_state = self._restore_state_from_destination()
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 1588, in _restore_state_from_destination
state = load_pipeline_state_from_destination(self.pipeline_name, job_client)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\state_sync.py", line 130, in load_pipeline_state_from_destination
state = client.get_stored_state(pipeline_name)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\destinations\impl\filesystem\filesystem.py", line 730, in get_stored_state
for filepath, fileparts in self._list_dlt_table_files(self.schema.state_table_name):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\destinations\impl\filesystem\filesystem.py", line 630, in _list_dlt_table_files
if not self.fs_client.exists(self.pathlib.join(dirname, INIT_FILE_NAME)):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\fsspec\asyn.py", line 118, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\fsspec\asyn.py", line 103, in sync
raise return_result
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\fsspec\asyn.py", line 56, in _runner
result[0] = await coro
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 1093, in _exists
await self._info(path, bucket, key, version_id=version_id)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 1445, in _info
out = await self._call_s3(
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 371, in _call_s3
return await _error_wrapper(
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 146, in _error_wrapper
raise err
OSError: [Errno 22] Bad Request
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\temp.py", line 26, in
info = pipeline.run(resource)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 228, in _wrap
step_info = f(self, *args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 277, in _wrap
return f(self, *args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 710, in run
self._sync_destination(destination, staging, dataset_name)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 182, in _wrap
rv = f(self, *args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 879, in _sync_destination
raise PipelineStepFailed(self, "sync", None, ex, None) from ex
dlt.pipeline.exceptions.PipelineStepFailed: Pipeline execution failed at `step=sync` with exception:
[Errno 22] Bad Request
```
### Expected behavior
The parquet file should be created in the bucket configured.
### Steps to reproduce
Start a fresh python interpreter, install dlt[filesystem,parquet] and run the code described. I'm using this pyproject.toml:
```
[project]
name = "simple-data-platform"
version = "0.1.0"
description = "Add your description here"
requires-python = ">=3.10"
dependencies = [
"dlt[filesystem,parquet]==1.13.0",
"python-dotenv>=1.1.1",
]
```
### Operating system
Windows
### Runtime environment
Local
### Python version
3.10
### dlt data source
python data source
### dlt destination
Filesystem & buckets
### Other deployment details
_No response_
### Additional information
_No response_
Contributor guide
Research direction
Reproduce the issue with the provided dlt[filesystem,parquet] setup and BUCKET_URL=s3://... configuration. Start in dlt/destinations/impl/filesystem/filesystem.py at get_stored_state and _list_dlt_table_files, then trace the S3 exists check shown in the traceback. Done means the pipeline sync completes and creates the parquet file in the configured bucket.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100