Blobbers can become unresponsive when there are large number of files/directories.
- Linguagem predominante
- Go
- Estrelas
- 30
- Forks
- 32
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
I came across _list-all_ subcommand in zboxcli which uses **getRemoteFilesAndDirs** function: [https://github.com/0chain/gosdk/blob/master/zboxcore/sdk/sync.go#L44](url)
What it does is, it requests blobbers recursively to get list of files/directories and further traverses inside each child directories and so on till end. So if there are say 100 subdirectories then it will make atleast 100 such requests.
There is another option to request for ObjectTree from blobbers i.e. to call http request to blobbers as given in the doc: [https://api.0chain.net/#402a1367-2f35-430b-9eaa-42917ead886b](url)
So for instance if I send request for retrieving ObjectTree for root path then it will return the json response of whole file hierarchy in that respective allocation.
Above call is fine if there are smaller number of files but we need to consider that an allocation can contain thousands of such files. For about 5 directories and 5 files the response size was about 60KB so for large number of files it will be large sized response making blobber busy to serve request for certain amount of time as metadata can be of for example; 20MB which obviously stalls the blobber.
And above is just for single allocation scenario. Blobbers however are not confined to single allocation and there can be multitude of clients requests.
So the solution can be to provide paginated response or partial tree response(say we only provide few levels of tree depth in response).
There are two other options i.e. ObjectPath and ReferencePath requests. However both can grow larger in response size and have same issue as ObjectTree requests.
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
The issue is about the zboxcli's list-all subcommand and the getRemoteFilesAndDirs function in sync.go. The current recursive approach makes many requests; the proposed solution is pagination or partial tree depth for ObjectTree API calls. Start by examining the sync.go file and the blobber's HTTP API documentation to understand the current metadata retrieval. Then design a paginated or depth-limited response mechanism that avoids large payloads and blobber stalls.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- go
- Domínio
- api, backend, performance
- Tipo de issue
- Funcionalidade
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Estagnada
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 35/100