apache / apache/cloudstack

Storage bandwidth is wasted during template uploads/imports

未關閉
#5,697 6 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
component:secondary-storage component:templates no-issue-activity status:stale type:improvement
主要語言
Java
星號
3.1k
分支
1.4k
平均合併
6 天 19 小時
30 天內合併 PR
32

描述

At the moment when a template is being imported (via url or upload) there is a phase called "Installing template" or something similar. From what I noticed this phase calculates a hash and saves it to the database by reading the downloaded file over the storage network.
During this phase the template is not usable and I believe that should not be the case. I understand that the hash has a purpose and I am not saying it should be removed but I believe this should be an optional task that should be done in background.

In the following examples I do not consider disk speeds, just network speed/bandwidth.
For normal templates this is not necessarily noticeable. Consider this scenario (best case):
Template size: 4GB
Ingress bandwidth: 100 mb/s
Storage bandwidth: 1 gb/s
Download time required: 5.45 seconds
Installing template time required: ~0.6 seconds

Most templates (not ISOs) however are considerably larger, some could be even up to 500 GB (I do have a few templates that I have to import with very large sizes).
In such a case, installing template time required would be about 75 seconds (best case).

During this time (installing template time) the whole bandwidth available for the storage network (if it even is on a separate NIC) would be used up by this process resulting in bad performance for the cluster.

Ways to fix this would be:
1. either compute the hash as the transfer is happening.
2. make it optional (maybe even opt-in) and do it in background only (maybe even limit the bandwidth used for this)

貢獻指南

開啟貢獻指南

研究方向

先追蹤「Installing template」階段周圍的模板 URL/上傳匯入路徑,然後找出下載的檔案在哪裡從儲存空間重新讀取,以及其雜湊在哪裡儲存。比較傳輸與從儲存空間讀取的步驟。完成的標準是:模板可用性不會被一次獨立的完整儲存空間讀取所阻塞,同時必要的雜湊仍然正確,且已定義所提議的背景處理或頻寬行為。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
java
領域
cloud, infrastructure, performance
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。