nspcc-dev / nspcc-dev/neofs-node

Improve combined fstree format

Open
#3,418 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement I4 neofs-storage performance S3 U3
Dominant language
Go
Stars
38
Forks
51
Avg merge
1d 20h
Merged PRs (30d)
38

Description

Is your feature request related to a problem? Please describe.

I'm always frustrated when reads from FSTree take more time than they could in case of combined format. We do not have any OID: [offset, size] map there, so reader has to Seek() through to get the object he needs. For 128 objects in a pack it could take some time.

Describe the solution you'd like

A table can be stored at the end of the file with this data, it's trivial to do for writer and then reader can:

  • open file, read a chunk
  • try the first object, try others already read, if found --- OK
  • if not, seek to length-MAX_TABLE_SIZE of the file
  • read data
  • find the number of entries stored in the last bytes
  • jump backwards through the data to find the object needed
  • Seek to it directly, read

This works for files generated on-the-fly from processed PUT requests. But if we have a write cache and use direct batched writing this can be further optimized by placing the table at the beginning of the file (format should support both). Then reader can get and use it immediately after initial read.

Describe alternatives you've considered

Not known.

Additional context

https://github.com/nspcc-dev/neofs-node/pull/2814
https://github.com/nspcc-dev/neofs-node/issues/2925

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the linked PR #2814 and issue #2925, then trace the combined FSTree reader and writer described in the issue. Determine how the format can represent a lookup table at the file start or end, and validate that readers can locate objects without scanning the pack.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.