0chain / 0chain/gosdk

Blobbers can become unresponsive when there are large number of files/directories.

未關閉
#117 7 則留言 0 個 reaction 已指派 1 人 已指派給 @lpoli 在 GitHub 檢視
post-mainnet refactor
主要語言
Go
星號
30
分支
32
PR 合併指標
30 天內沒有已合併 PR

描述

I came across _list-all_ subcommand in zboxcli which uses **getRemoteFilesAndDirs** function: [https://github.com/0chain/gosdk/blob/master/zboxcore/sdk/sync.go#L44](url)

What it does is, it requests blobbers recursively to get list of files/directories and further traverses inside each child directories and so on till end. So if there are say 100 subdirectories then it will make atleast 100 such requests.

There is another option to request for ObjectTree from blobbers i.e. to call http request to blobbers as given in the doc: [https://api.0chain.net/#402a1367-2f35-430b-9eaa-42917ead886b](url)
So for instance if I send request for retrieving ObjectTree for root path then it will return the json response of whole file hierarchy in that respective allocation.

Above call is fine if there are smaller number of files but we need to consider that an allocation can contain thousands of such files. For about 5 directories and 5 files the response size was about 60KB so for large number of files it will be large sized response making blobber busy to serve request for certain amount of time as metadata can be of for example; 20MB which obviously stalls the blobber.

And above is just for single allocation scenario. Blobbers however are not confined to single allocation and there can be multitude of clients requests.

So the solution can be to provide paginated response or partial tree response(say we only provide few levels of tree depth in response).
There are two other options i.e. ObjectPath and ReferencePath requests. However both can grow larger in response size and have same issue as ObjectTree requests.

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

The issue is about the zboxcli's list-all subcommand and the getRemoteFilesAndDirs function in sync.go. The current recursive approach makes many requests; the proposed solution is pagination or partial tree depth for ObjectTree API calls. Start by examining the sync.go file and the blobber's HTTP API documentation to understand the current metadata retrieval. Then design a paginated or depth-limited response mechanism that avoids large payloads and blobber stalls.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
go
領域
api, backend, performance
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。