ipfs / ipfs/kubo

RFC: Switch to using Inode like datastructures in the blockstore

Open
#5,528 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

Our current blockstore has complete lack of support for any sort of metadata. I believe fairly strongly that a lot of problems can be made a lot easier if we support it. One very good way to do this is to introduce the concept of a `BlockInfo` which adds a lawyer of indirection in a similar manor of unix inodes.

A possible API:
```go
type Blockstore interface {
Has(mh.Multihash) (bool, error)

Get(cid.Cid) (*BlockInfo, error)

// Update atomically updates a block info. If the origData field
// of the BlockInfo is not nil the value is retrieved and compared
// to this value, if values match the store is updated with the
// new info in BlockInfo, if the value does not match then the
// store is not updated. If the origData field is empty and the
// block already exists than the block is not updated. The first
// value returned is true if the block was updated. The second
// value is the new blockinfo as it exists in the Store. The final
// value returnd is an error.
Update(val *BlockInfo) (bool, *BlockInfo, error)

// Overright updates the block info with no regard to the existing
// value. Use with care
Overright(val *BlockInfo) error

// TODO UpdateMany, will lock then first retrieve all keys and
// compare, after that it will update the store according to the
// semantic described in Update in a batch update

Delete(mh.Multihash) error

// AllKeysChan returns a channel from which
// the CIDs in the Blockstore can be read. It should respect
// the given context, closing the channel if it becomes Done.
AllKeysChan(ctx context.Context) (<-chan mh.Multihash, error)

// HashOnRead specifies if every read block should be
// rehashed to make sure it matches its CID.
HashOnRead(enabled bool)
}

type BlockInfo struct {
Cid cid.Cid
Accessed time.Time // can be an approximate value for performance reason
Created time.Time
PinCount int // Delete will return an error if PinCount > 0
Block BlockLink
Attr []BlockAttr
origData []byte // orignal data as retrieved from the blockstore
}

type BlockLink interface {
// Get and contruct a block using the provided Cid
func Get(cid.Cid) (Block, error)
// Size returns the size of the block, it should never error
func Size() int
}

type InlineBlock struct { //implements BlockLink
data []byte
}

type NormalBlock struct { //implements BlockLink
size int
// other fields, if necessary to point to the location of the
// block
}

type ExternalBlock struct { //implements BlockLink
url string
offset int
size int
}

type BlockAttr struct {
Key string
Value interface{}
}
```

Advantages to using this API:

* Maintains pin reference counts to speed up pin operations
* Allows for maianting last access time for more intelligent garbage collection
* Allows for a hybrid storage approach where small blocks are stored in the database while larger blocks are stored via other means, for example using flatfs for additional flexibility and ability to take advantage of filesystem level deduplication
* Builtin support filestore/urlstore support
* Provide means to store additional useful attributes with the block itself rather then in a different data-structure
* In general provides a lot more flexibility and should make other things easier down the road

Challenges:

The backing datastore needs to allow for fast retrieval and updating of keys

---

I agree this will be a major change but I think it is something worth considering. If no one else wants to attempt this I will be happy lead the effort.

_(Note: I see this as an actionable action possible specific to go-ipfs (at least at first), if there is a better place to file this please let me know.)_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.