0chain / 0chain/gosdk

Blobbers can become unresponsive when there are large number of files/directories.

Aberta
#117 7 comentários 0 reações 1 responsável Atribuída a @lpoli Ver no GitHub
post-mainnet refactor
Linguagem predominante
Go
Estrelas
30
Forks
32
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

I came across _list-all_ subcommand in zboxcli which uses **getRemoteFilesAndDirs** function: [https://github.com/0chain/gosdk/blob/master/zboxcore/sdk/sync.go#L44](url)

What it does is, it requests blobbers recursively to get list of files/directories and further traverses inside each child directories and so on till end. So if there are say 100 subdirectories then it will make atleast 100 such requests.

There is another option to request for ObjectTree from blobbers i.e. to call http request to blobbers as given in the doc: [https://api.0chain.net/#402a1367-2f35-430b-9eaa-42917ead886b](url)
So for instance if I send request for retrieving ObjectTree for root path then it will return the json response of whole file hierarchy in that respective allocation.

Above call is fine if there are smaller number of files but we need to consider that an allocation can contain thousands of such files. For about 5 directories and 5 files the response size was about 60KB so for large number of files it will be large sized response making blobber busy to serve request for certain amount of time as metadata can be of for example; 20MB which obviously stalls the blobber.

And above is just for single allocation scenario. Blobbers however are not confined to single allocation and there can be multitude of clients requests.

So the solution can be to provide paginated response or partial tree response(say we only provide few levels of tree depth in response).
There are two other options i.e. ObjectPath and ReferencePath requests. However both can grow larger in response size and have same issue as ObjectTree requests.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

The issue is about the zboxcli's list-all subcommand and the getRemoteFilesAndDirs function in sync.go. The current recursive approach makes many requests; the proposed solution is pagination or partial tree depth for ObjectTree API calls. Start by examining the sync.go file and the blobber's HTTP API documentation to understand the current metadata retrieval. Then design a paginated or depth-limited response mechanism that avoids large payloads and blobber stalls.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
go
Domínio
api, backend, performance
Tipo de issue
Funcionalidade
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Estagnada
Clareza
Razoavelmente clara
Facilidade para iniciantes
35/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.