Blobbers can become unresponsive when there are large number of files/directories.
- 主要語言
- Go
- 星號
- 30
- 分支
- 32
- PR 合併指標
- 30 天內沒有已合併 PR
描述
I came across _list-all_ subcommand in zboxcli which uses **getRemoteFilesAndDirs** function: [https://github.com/0chain/gosdk/blob/master/zboxcore/sdk/sync.go#L44](url)
What it does is, it requests blobbers recursively to get list of files/directories and further traverses inside each child directories and so on till end. So if there are say 100 subdirectories then it will make atleast 100 such requests.
There is another option to request for ObjectTree from blobbers i.e. to call http request to blobbers as given in the doc: [https://api.0chain.net/#402a1367-2f35-430b-9eaa-42917ead886b](url)
So for instance if I send request for retrieving ObjectTree for root path then it will return the json response of whole file hierarchy in that respective allocation.
Above call is fine if there are smaller number of files but we need to consider that an allocation can contain thousands of such files. For about 5 directories and 5 files the response size was about 60KB so for large number of files it will be large sized response making blobber busy to serve request for certain amount of time as metadata can be of for example; 20MB which obviously stalls the blobber.
And above is just for single allocation scenario. Blobbers however are not confined to single allocation and there can be multitude of clients requests.
So the solution can be to provide paginated response or partial tree response(say we only provide few levels of tree depth in response).
There are two other options i.e. ObjectPath and ReferencePath requests. However both can grow larger in response size and have same issue as ObjectTree requests.
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
The issue is about the zboxcli's list-all subcommand and the getRemoteFilesAndDirs function in sync.go. The current recursive approach makes many requests; the proposed solution is pagination or partial tree depth for ObjectTree API calls. Start by examining the sync.go file and the blobber's HTTP API documentation to understand the current metadata retrieval. Then design a paginated or depth-limited response mechanism that avoids large payloads and blobber stalls.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- go
- 領域
- api, backend, performance
- Issue 類型
- 功能
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100