grafana / grafana/pyroscope

Create config setting to customize an environment for querying

Open
#2,796 1 comment 0 reactions 2 assignees Claimed by @simonswine View on GitHub
backend enhancement performance ux
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

## The problem

When we query profiles for applications that are very large, it's pertinent to add additional context to the query to scope it down to query a more realistic result set. There's a couple different strategies that can achieve this end:

1. Use relabeling to synthesize this label `service_name=/`. The `namespace` prefix ensures we restrict the query to a reasonable subset of profiles (this is what we do today).
2. Manually add another label to the query to ensure the result set is manageable. E.g.

```
{namespace="", service_name=""}
```

Both of these options work, but have some short-comings. In option 1, we are forcing users to use relabel rules to make Pyroscope usable at scale. In option 2, we are requiring that users have an understanding that they need specific labels in order to create a query that is realistic to execute.

The long and short is, both options reduce Pyroscope's out-of-the-box usability. There's additional know-how and semi-arcane knowledge that is required before Pyroscope can be used at scale.

## A solution

To help bridge this usability gap, we can create a mandatory label that is used to qualify all queries. This label can be configurable through our configuration file and we can provide a reasonable default value to allow users who aren't dealing with large scale applications to avoid having to worry about such details.

To better illustrate this point, we could have two additional configuration values:

- `querier.environment_label.name` - Sets the mandatory label name for every query (example: `"namespace"`)
- `querier.environment_label.default_value` - Sets the default label value if the mandatory label is not provided (example: `"profiles-ops-001"` or `""`)

This would require all queries to either:

- Provide a `namespace=` value with every query
- Use `namespace=` if no `namespace` label was provided in the query

### Alternative

As Cyril [pointed out](https://github.com/grafana/pyroscope/issues/2796#issuecomment-1838084350), this could be managed entirely within the UI. This should help reduce complexity in the Pyroscope server configuration.

## Considerations

This should help improve a usability concern of Pyroscope and allow every users to customize Pyroscope queries to respect their environments, without having to go through extra hoops of relabeling or knowing specific labels that always need to be added (but aren't required) to reduce the result set to a realistic size.

That said, there are concerns about whether this approach is flexible enough. Certainly there may be some customers who have more complicated use cases which this will not full cover. Additionally, we need to consider the UX when designing queries. We need to make it obvious that a specific label must be specified or a default value will be used. We also should accommodate users wanting to opt out of this feature entirely.
```[tasklist]
### Tasks
- [ ] https://github.com/grafana/pyroscope/pull/2711
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.