Problem uploading empty files with upload_object
- Vorherrschende Sprache
- Python
- Sterne
- 2.1k
- Forks
- 931
- Ø Merge
- 1 T. 2 Std.
- Gemergte PRs (30 T.)
- 4
Beschreibung
## Summary
Using `upload_object` to upload an empty file returns response:
`400 - Bad Request: too many Host headers`
## Detailed Information
Python 3.6.8
Libcloud: 3.3.1
OS: rhel 7.9
I use libcloud to upload objects to 3 providers - aws s3, minio & netapp storageGrid, I use the s3 driver for all of them. `upload_object` function works fine except when I try to upload an empty file (I sometimes need to store just meta-data with a specific object name for later use). When trying to upload an empty file I get the response:
```
status: 400
reason: 'Bad Request: too many Host headers'
```
If I add anything to the file's content it uploads successfully with the meta-data, so its not a problem with permissions/path in s3 or something like that. I also tried without the meta-data, but that didn't work either.
I tried using `upload_object_via_stream` and found the following code works:
```
with io.BytesIO() as iterator:
driver.upload_object_via_stream(iterator=iterator,
container=container,
object_name=objName,
extra=extra)
```
This code uploads a 0 bytes file successfully to aws s3 & minio, but not to netapp's storageGrid (so unfortunately I can't use it).
I think it's a limitation of storageGrid's multi-part upload and not some other storage restriction (we have existing empty objects on storageGrid, but those were uploaded using jclouds).
I think using `upload_object` will be the best solution for all 3 providers, but I don't understand why it's not uploading empty files and would appreciate help with this. Thanks!
### Reproduce
1. `touch ~/test-empty.txt`
2. Use `upload_object` to upload test-empty.txt
Beitragsleitfaden
Rechercherichtung
Die gemeldeten Einstiegspunkte sind upload_object und upload_object_via_stream; beginne damit, den Fall einer leeren Datei mit dem S3-Treiber zu reproduzieren und das Verhalten zwischen AWS S3, MinIO und NetApp StorageGRID zu vergleichen. Da im Issue keine Quelldatei oder kein Test genannt wird, ermittle zuerst die Implementierung des Uploads und die zugehörigen Tests. Als erledigt gilt die Aufgabe, wenn upload_object leere Dateien mit Metadaten verarbeitet, ohne doppelte Host-Header zu erzeugen oder die Kompatibilität mit Providern zu beeinträchtigen.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- aws, python
- Bereich
- cloud
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 35/100