dlt-hub / dlt-hub/dlt

S3 destination giving botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

Open
#2,872 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

1.13.0

### Describe the problem

I'm trying to run this simple code using a S3 bucket as destination. I'm using environment variables to put the config parameters.

```
import dlt
from dotenv import load_dotenv

load_dotenv()

def hello_world_resource():
@dlt.resource(name="hello_world", write_disposition="replace", file_format="parquet")
def resource():
data = [
{"message": "Hello World", "idioma": "english", "id": 1},
]

for item in data:
yield item

return resource()

pipeline = dlt.pipeline(
pipeline_name="hello_world_pipeline",
destination="filesystem",
dataset_name="hello_world_dataset",
)

if __name__ == "__main__":
resource = hello_world_resource()
info = pipeline.run(resource)

```

When I use `BUCKET_URL=somefolder`, this code works fine.

When I use `BUCKET_URL=s3://mybucket-that-exists` I'm getting the error below. My .env file is configured as follows:
```
AWS_ACCESS_KEY_ID="secret"
AWS_SECRET_ACCESS_KEY="secret"
AWS_REGION="us-east-1"
BUCKET_URL='s3://mybucket-that-exists'
```
Error:
```
Traceback (most recent call last):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 114, in _error_wrapper
return await func(*args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\aiobotocore\client.py", line 412, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 783, in _sync_destination
remote_state = self._restore_state_from_destination()
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 1588, in _restore_state_from_destination
state = load_pipeline_state_from_destination(self.pipeline_name, job_client)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\state_sync.py", line 130, in load_pipeline_state_from_destination
state = client.get_stored_state(pipeline_name)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\destinations\impl\filesystem\filesystem.py", line 730, in get_stored_state
for filepath, fileparts in self._list_dlt_table_files(self.schema.state_table_name):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\destinations\impl\filesystem\filesystem.py", line 630, in _list_dlt_table_files
if not self.fs_client.exists(self.pathlib.join(dirname, INIT_FILE_NAME)):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\fsspec\asyn.py", line 118, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\fsspec\asyn.py", line 103, in sync
raise return_result
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\fsspec\asyn.py", line 56, in _runner
result[0] = await coro
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 1093, in _exists
await self._info(path, bucket, key, version_id=version_id)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 1445, in _info
out = await self._call_s3(
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 371, in _call_s3
return await _error_wrapper(
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\s3fs\core.py", line 146, in _error_wrapper
raise err
OSError: [Errno 22] Bad Request

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\temp.py", line 26, in
info = pipeline.run(resource)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 228, in _wrap
step_info = f(self, *args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 277, in _wrap
return f(self, *args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 710, in run
self._sync_destination(destination, staging, dataset_name)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 182, in _wrap
rv = f(self, *args, **kwargs)
File "C:\Users\victor.outtes_a3data\PycharmProjects\simple-data-platform\.venv\lib\site-packages\dlt\pipeline\pipeline.py", line 879, in _sync_destination
raise PipelineStepFailed(self, "sync", None, ex, None) from ex
dlt.pipeline.exceptions.PipelineStepFailed: Pipeline execution failed at `step=sync` with exception:

[Errno 22] Bad Request
```

### Expected behavior

The parquet file should be created in the bucket configured.

### Steps to reproduce

Start a fresh python interpreter, install dlt[filesystem,parquet] and run the code described. I'm using this pyproject.toml:
```
[project]
name = "simple-data-platform"
version = "0.1.0"
description = "Add your description here"
requires-python = ">=3.10"
dependencies = [
"dlt[filesystem,parquet]==1.13.0",
"python-dotenv>=1.1.1",
]
```

### Operating system

Windows

### Runtime environment

Local

### Python version

3.10

### dlt data source

python data source

### dlt destination

Filesystem & buckets

### Other deployment details

_No response_

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the provided dlt[filesystem,parquet] setup and BUCKET_URL=s3://... configuration. Start in dlt/destinations/impl/filesystem/filesystem.py at get_stored_state and _list_dlt_table_files, then trace the S3 exists check shown in the traceback. Done means the pipeline sync completes and creates the parquet file in the configured bucket.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.