awslabs / awslabs/aws-crt-python

awscrt is ~5x slower at checking file existence via HEAD operations when the files do not exist vs. when they do

Open
#699 2 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
Python
Stars
100
Forks
58
Avg merge
13h 51m
Merged PRs (30d)
2

Description

### Describe the bug

`make_request(operation_name="HeadObject")` in `awscrt.s3.S3Client` is ~5x slower at checking file existence when the files do not exist vs. when they do.

It seems like awscrt is spending significantly more time processing the 404 (I get a `AWS_ERROR_S3_INVALID_RESPONSE_STATUS` exception with `status_code` 404 for the HEAD operation when the file does not exist) than when it exists.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

I expect HEAD operations to perform about the same speed regardless of whether the file in S3 exists or not.

### Current Behavior

I wrote a sample script to perform 1000 HEAD against an S3 server. When the files existed, awscrt took 2.44s to check them all. When the files do not exist, it took 12.23s. About 5X slower. Running this script multiple times yielded similar results.

### Reproduction Steps

See attached test script. Make sure to fill in:
```
# Fill these in ...
server = "server.example.com"
bucket = "my-bucket"
prefix = "test-scratch/file_"
```
and ensure `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set in the environment (or use another authentication mechanism).

### Possible Solution

Handle 404 errors for HEAD operations similarly to the success case (status 200).

### Additional Information/Context

_No response_

### aws-crt-python version used

awscrt==0.28.3

### Python version used

Python 3.13.7

### Operating System and version

macOS 15.6.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.