grafana / grafana/pyroscope

Prototype ingestor-less replication-less architecture

Open
#3,192 0 comments 4 reactions 0 assignees View on GitHub
backend
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Is your feature request related to a problem? Please describe.

Currently at large scale Pyroscope struggles with a few issues:
* compaction
* is using a lot of resources, particularly ram
* this is mostly because of the need for deduplication of data, which is caused by replication
* replication (x3)
* uses a lot of resources (cup, disk, ram)
* is hard to maintain
* leads to complex bugs
* reduces read performance due to the need for deduplication of data
* reads and writes are not isolated so spike in reads often affects writes

#### Describe the solution you'd like

We could do the following:
* remove ingesters
* make it so that distributors create small blocks in memory and flush them to object storage
* remove deduplication code from read path and compaction
* tweak compaction to work with increased number of blocks
* tweak read path to work with increased number of blocks

#### Concerns / Risks

These are not blockers, but rather a list of things that might derail the project, so we should make sure we keep these concerns in mind and address these early:

* increased object storage write costs
* reduced query performance due to too many small blocks
* running into some unforeseen performance limitations of underlying object storage

#### Acceptance Criteria

* [ ] it should work with traffic in ops
* [ ] costs should not go up. it is fine if we have to exchange reduced ingesters costs with increased queriers cost though, because we can address this later
* [ ] performance should not go down

#### Timeline / Staffing

Great news is that we already have all of the components built for this and so this project becomes a lot more about moving things around and tweaking the system rather than building new stuff.

I think if we split up the task we could get a good working prototype in about 4 weeks and 3 people. I think @kolesnikovae should lead this project.

Assuming the project succeeds we could then spend another 4 weeks doing migrations / further tweaking algorithms to work with the new system.

#### Outcome

The main thing is that the system becomes simpler. To elaborate on that, these changes would:
* improve maintainability of the system
* reduce toil
* significantly reduce tco
* improve read performance

#### Additional context

All credit for this idea should go to @kolesnikovae — I'm just trying to document the proposed solution. Also, this is somewhat of a meta-issue. I imagine a lot of other existing issues could be steps towards implementation of this project.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.