Storage bandwidth is wasted during template uploads/imports
- Vorherrschende Sprache
- Java
- Sterne
- 3.1k
- Forks
- 1.4k
- Ø Merge
- 6 T. 19 Std.
- Gemergte PRs (30 T.)
- 32
Beschreibung
At the moment when a template is being imported (via url or upload) there is a phase called "Installing template" or something similar. From what I noticed this phase calculates a hash and saves it to the database by reading the downloaded file over the storage network.
During this phase the template is not usable and I believe that should not be the case. I understand that the hash has a purpose and I am not saying it should be removed but I believe this should be an optional task that should be done in background.
In the following examples I do not consider disk speeds, just network speed/bandwidth.
For normal templates this is not necessarily noticeable. Consider this scenario (best case):
Template size: 4GB
Ingress bandwidth: 100 mb/s
Storage bandwidth: 1 gb/s
Download time required: 5.45 seconds
Installing template time required: ~0.6 seconds
Most templates (not ISOs) however are considerably larger, some could be even up to 500 GB (I do have a few templates that I have to import with very large sizes).
In such a case, installing template time required would be about 75 seconds (best case).
During this time (installing template time) the whole bandwidth available for the storage network (if it even is on a separate NIC) would be used up by this process resulting in bad performance for the cluster.
Ways to fix this would be:
1. either compute the hash as the transfer is happening.
2. make it optional (maybe even opt-in) and do it in background only (maybe even limit the bandwidth used for this)
Beitragsleitfaden
Rechercherichtung
Beginne damit, den Importpfad für Template-URL/-Upload rund um die Phase „Installing template“ nachzuverfolgen, und ermittle anschließend, wo die heruntergeladene Datei erneut aus dem Speicher gelesen wird und wo ihr Hash gespeichert wird. Vergleiche die Schritte für Übertragung und Lesen aus dem Speicher. Als abgeschlossen gilt die Arbeit, wenn die Verfügbarkeit des Templates nicht durch einen separaten vollständigen Lesevorgang aus dem Speicher blockiert wird, der erforderliche Hash weiterhin korrekt ist und das vorgeschlagene Verhalten für Hintergrundverarbeitung oder Bandbreite definiert ist.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java
- Bereich
- cloud, infrastructure, performance
- Issue-Typ
- Feature
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 45/100