File listing Optimization Proposal

Retrieving file listings, and in particular `list-all` will increasingly become more resource hungry as the number of files and directories in an allocation grow

Separate to having a folders-only option, here is an idea how file listings could be done more efficiently for large numbers of files, (compared with apparent current method that client requests lists from all blobbers and seeks sufficient consensus majority to consider a correct listing):-

* instead of each blobber sending entire files listing, they create internal list with file paths and content hash only (consistent between blobbers)
* This list is sorted
* A hash of this list is generated (consistent between blobbers)
* This list (referenced by it's hash) is saved (temporarily)
* The hash is what is returned to client initially
* The client only has to compare majority (consensus) of hashes to ensure file listing correctness.
* Client only has to actually retrieve file listing from random one of matching blobbers
* This can be done with pagination, solving consistency issue when time elapses between pages
* In fact pagination requests can be split by Blobbers so blobber1 page 1, blobber2 page 2 etc.
* Cached list can be retained at least for a short while for pagination requests but actually remains valid until write operation performed on allocation. So perhaps flagged as stale as soon as a CRUD operation performed then stale listing removed after say 1 minute.
* This could form a secondary method of file listing at a certain threshold, perhaps for any more than say, 1,000 files or wherever pagination would be decided.

Examina la lógica de listado de archivos en la base de código del blobber, probablemente en un manejador o servicio relacionado con las operaciones de archivos de asignación. Comprende el mecanismo de consenso actual para los listados. La propuesta implica crear una lista hash y ordenada de rutas de archivos y hashes de contenido, almacenarla en caché e implementar la paginación. Comienza por encontrar dónde se implementa actualmente list-all y examina las estructuras de datos utilizadas para los metadatos de archivos.

Escrito por el modelo de indexación a partir del texto del issue.