Storage bandwidth is wasted during template uploads/imports
- Langage dominant
- Java
- Étoiles
- 3.1k
- Forks
- 1.4k
- Merge moyen
- 6 j 19 h
- PR mergées (30 j)
- 32
Description
At the moment when a template is being imported (via url or upload) there is a phase called "Installing template" or something similar. From what I noticed this phase calculates a hash and saves it to the database by reading the downloaded file over the storage network.
During this phase the template is not usable and I believe that should not be the case. I understand that the hash has a purpose and I am not saying it should be removed but I believe this should be an optional task that should be done in background.
In the following examples I do not consider disk speeds, just network speed/bandwidth.
For normal templates this is not necessarily noticeable. Consider this scenario (best case):
Template size: 4GB
Ingress bandwidth: 100 mb/s
Storage bandwidth: 1 gb/s
Download time required: 5.45 seconds
Installing template time required: ~0.6 seconds
Most templates (not ISOs) however are considerably larger, some could be even up to 500 GB (I do have a few templates that I have to import with very large sizes).
In such a case, installing template time required would be about 75 seconds (best case).
During this time (installing template time) the whole bandwidth available for the storage network (if it even is on a separate NIC) would be used up by this process resulting in bad performance for the cluster.
Ways to fix this would be:
1. either compute the hash as the transfer is happening.
2. make it optional (maybe even opt-in) and do it in background only (maybe even limit the bandwidth used for this)
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par suivre le chemin d’importation de l’URL/de l’upload du template autour de la phase « Installing template », puis identifiez où le fichier téléchargé est relu depuis le stockage et où son hash est enregistré. Comparez les étapes de transfert et de lecture depuis le stockage. Le travail est considéré comme terminé lorsque la disponibilité du template n’est pas bloquée par une lecture complète séparée depuis le stockage, que le hash requis reste correct et que le comportement proposé en arrière-plan ou en matière de bande passante est défini.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- cloud, infrastructure, performance
- Type d'issue
- Fonctionnalité
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- Plutôt claire
- Accessibilité débutants
- 45/100