cockroachdb / cockroachdb/pebble

using columnar block format and keySchemas for user-level keys

Open
#4,380 1 comment 0 reactions 0 assignees View on GitHub
A-storage C-question no-issue-activity O-community T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

I've been exploring the new columnar format and KeySchemas. My hope is to leverage the KeySchemas for my user-level keys for objects and secondary indexes so that I may benefit from the efficiencies provided by columnar block format and faster scans. I hope too it'll benefit on faster writes in some way or faster compaction, with reduced storage requirements, but who knows yet :)

we've built a light db table and query layer on top of pebble which we've been using in prod for over a year with great success: https://github.com/go-bond/bond – and we're hoping to extend it and provide additional performance gains through the use of the new columnar format + keyschemas.

my questions is:
1. does it make sense to use the columnar format + KeySchemas for defining the schema for user-level keys? Or is there little benefit as the prefixes are already compressed, and scans / writes will be the same in performance?

if it does make sense, in the case of our use.. we have "tables" like a traditional database which uses a `tableID` prefix for all keys, and then we define the primary key in parts. We also define the `/tableID/indexID/` for our secondary indexes of an inverted index for querying specific fields. Each table has a different "keySchema" because we may define the primaryKey slightly differently depending on how we define the object to be "unique". We could of course do something like hash the contents and use that as the primary key id, with like blake2b or blake3, but it would add overhead to compute the primary id for each query, but perhaps this overhead is better then having long keys in pebble, or having a different keySchema to represent the table of rows (objects) by key. That may work for primary table keys, but it wouldn't work for secondary indexes which vary in the set of components (aka columns) we'd use for the search.

so, follow up question:

2. I understand a sstable has a KeySchema object on it as well, and so different sstables may have their own keyschema (excellent), thanks to https://github.com/cockroachdb/pebble/pull/4057 – but, how can I use this in practice on a single db instance of pebble? Ideally for any Set/Put of a key, I would give a hint to pebble on the keySchema name, and then it would essentially partition my data within the db across different sstables. This would be helpful, as of course my secondary indexes will also have their own keySchema.

please lmk if any of this makes sense.

thank you again for the amazing product, and I appreciate to see pebble continuing to evolve so well

Jira issue: PEBBLE-350

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.