nspcc-dev / nspcc-dev/neofs-node
Improve combined fstree format
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 38
- Forks
- 51
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 38
Description
Is your feature request related to a problem? Please describe.
I'm always frustrated when reads from FSTree take more time than they could in case of combined format. We do not have any OID: [offset, size] map there, so reader has to Seek() through to get the object he needs. For 128 objects in a pack it could take some time.
Describe the solution you'd like
A table can be stored at the end of the file with this data, it's trivial to do for writer and then reader can:
- open file, read a chunk
- try the first object, try others already read, if found --- OK
- if not, seek to length-MAX_TABLE_SIZE of the file
- read data
- find the number of entries stored in the last bytes
- jump backwards through the data to find the object needed
- Seek to it directly, read
This works for files generated on-the-fly from processed PUT requests. But if we have a write cache and use direct batched writing this can be further optimized by placing the table at the beginning of the file (format should support both). Then reader can get and use it immediately after initial read.
Describe alternatives you've considered
Not known.
Additional context
https://github.com/nspcc-dev/neofs-node/pull/2814
https://github.com/nspcc-dev/neofs-node/issues/2925
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the linked PR #2814 and issue #2925, then trace the combined FSTree reader and writer described in the issue. Determine how the format can represent a lookup table at the file start or end, and validate that readers can locate objects without scanning the pack.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100