awslabs / awslabs/aws-crt-python
awscrt is ~5x slower at checking file existence via HEAD operations when the files do not exist vs. when they do
- Dominant language
- Python
- Stars
- 100
- Forks
- 58
- Avg merge
- 13h 51m
- Merged PRs (30d)
- 2
Description
### Describe the bug
`make_request(operation_name="HeadObject")` in `awscrt.s3.S3Client` is ~5x slower at checking file existence when the files do not exist vs. when they do.
It seems like awscrt is spending significantly more time processing the 404 (I get a `AWS_ERROR_S3_INVALID_RESPONSE_STATUS` exception with `status_code` 404 for the HEAD operation when the file does not exist) than when it exists.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
I expect HEAD operations to perform about the same speed regardless of whether the file in S3 exists or not.
### Current Behavior
I wrote a sample script to perform 1000 HEAD against an S3 server. When the files existed, awscrt took 2.44s to check them all. When the files do not exist, it took 12.23s. About 5X slower. Running this script multiple times yielded similar results.
### Reproduction Steps
See attached test script. Make sure to fill in:
```
# Fill these in ...
server = "server.example.com"
bucket = "my-bucket"
prefix = "test-scratch/file_"
```
and ensure `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set in the environment (or use another authentication mechanism).
### Possible Solution
Handle 404 errors for HEAD operations similarly to the success case (status 200).
### Additional Information/Context
_No response_
### aws-crt-python version used
awscrt==0.28.3
### Python version used
Python 3.13.7
### Operating System and version
macOS 15.6.1
Contributor guide
Assessment
This issue has not been assessed yet.