apache / apache/libcloud

Problem uploading empty files with upload_object

Open
#1,562 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
931
Avg merge
1d 2h
Merged PRs (30d)
4

Description

## Summary

Using `upload_object` to upload an empty file returns response:
`400 - Bad Request: too many Host headers`

## Detailed Information

Python 3.6.8
Libcloud: 3.3.1
OS: rhel 7.9

I use libcloud to upload objects to 3 providers - aws s3, minio & netapp storageGrid, I use the s3 driver for all of them. `upload_object` function works fine except when I try to upload an empty file (I sometimes need to store just meta-data with a specific object name for later use). When trying to upload an empty file I get the response:
```
status: 400
reason: 'Bad Request: too many Host headers'
```

If I add anything to the file's content it uploads successfully with the meta-data, so its not a problem with permissions/path in s3 or something like that. I also tried without the meta-data, but that didn't work either.
I tried using `upload_object_via_stream` and found the following code works:
```
with io.BytesIO() as iterator:
driver.upload_object_via_stream(iterator=iterator,
container=container,
object_name=objName,
extra=extra)
```
This code uploads a 0 bytes file successfully to aws s3 & minio, but not to netapp's storageGrid (so unfortunately I can't use it).
I think it's a limitation of storageGrid's multi-part upload and not some other storage restriction (we have existing empty objects on storageGrid, but those were uploaded using jclouds).

I think using `upload_object` will be the best solution for all 3 providers, but I don't understand why it's not uploading empty files and would appreciate help with this. Thanks!

### Reproduce
1. `touch ~/test-empty.txt`
2. Use `upload_object` to upload test-empty.txt

Contributor guide

Open the contributing guide

Research direction

The reported entry points are upload_object and upload_object_via_stream; begin by reproducing the empty-file case with the S3 driver and compare behavior across AWS S3, MinIO, and NetApp StorageGRID. No source file or test is named in the issue, so locate the upload implementation and its tests first. Done means upload_object handles empty files with metadata without producing duplicate Host headers or breaking provider compatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.