cockroachdb / cockroachdb/pebble

add a compression analyzer facility

Open
#4,715 3 comments 0 reactions 1 assignee Claimed by @RaduBerinde View on GitHub
A-storage T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

This issue tracks adding a facility for analyzing data in real clusters. The goal is to get a good comparison between various compression algorithms and levels and use it to inform our suggested defaults or to add new adaptive compression algorithms.

We have two ways of doing this:
- online: we can sample blocks as they are written to or read from disk. For each sampled blocks, we run all experiments and retain statistics. This approach has the advantage of allowing us to accurately estimate CPU usage differences between algorithms within a specific workload. The disadvantage is that we can only produce data on clusters with versions that include this facility.
- "offline": we can add a CLI tool that looks at all relevant files from a store and samples blocks separately from any running process. This is easier to implement and provides a quicker way to obtain data, as a newer binary can be used just for this tool.

Jira issue: PEBBLE-442

Epic CRDB-49140

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.