grafana / grafana/pyroscope

Shutdown inactive tenants

Open
#2,047 0 comments 1 reaction 1 assignee Claimed by @kolesnikovae View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Describe the bug

We currently do not shut down tenants after they stop ingested or if they only query. This can be seen by errors like this;

```
level=error caller=phlaredb.go:157 ts=2023-06-15T09:34:35.028355532Z tenant=test-f msg="flushing head block failed" err="rename data/test-f/head/01H2Z6EDDK79F47B0YS41QQV42 data/test-f/local/01H2Z6EDDK79F47B0YS41QQV42: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:35:33.581804511Z tenant=test-e msg="flushing head block failed" err="rename data/test-e/head/01H2Z6G6KDX0PP7SK6A6K0PHSX data/test-e/local/01H2Z6G6KDX0PP7SK6A6K0PHSX: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:35:35.029651043Z tenant=test-f msg="flushing head block failed" err="rename data/test-f/head/01H2Z6G80KSC8RPGJJRFXXNMPW data/test-f/local/01H2Z6G80KSC8RPGJJRFXXNMPW: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:36:33.582905922Z tenant=test-e msg="flushing head block failed" err="rename data/test-e/head/01H2Z6J16DYPWGZ2T28HFSSWF3 data/test-e/local/01H2Z6J16DYPWGZ2T28HFSSWF3: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:36:35.029903716Z tenant=test-f msg="flushing head block failed" err="rename data/test-f/head/01H2Z6J2KNACMVXWJQJF2RW0FW data/test-f/local/01H2Z6J2KNACMVXWJQJF2RW0FW: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:37:33.583742876Z tenant=test-e msg="flushing head block failed" err="rename data/test-e/head/01H2Z6KVSE06TC0XDZ21RMQQ9T data/test-e/local/01H2Z6KVSE06TC0XDZ21RMQQ9T: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:37:35.03101748Z tenant=test-f msg="flushing head block failed" err="rename data/test-f/head/01H2Z6KX6N17XH6CWH2BD300NF data/test-f/local/01H2Z6KX6N17XH6CWH2BD300NF: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:38:33.584307462Z tenant=test-e msg="flushing head block failed" err="rename data/test-e/head/01H2Z6NPCEHQVC79TPXK989D5J data/test-e/local/01H2Z6NPCEHQVC79TPXK989D5J: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:38:35.032439564Z tenant=test-f msg="flushing head block failed" err="rename data/test-f/head/01H2Z6NQSPG2S5SJQ918A45TFH data/test-f/local/01H2Z6NQSPG2S5SJQ918A45TFH: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:39:33.585038255Z tenant=test-e msg="flushing head block failed" err="rename data/test-e/head/01H2Z6QGZFZYW1A4MXGWJT7DVT data/test-e/local/01H2Z6QGZFZYW1A4MXGWJT7DVT: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:39:35.032765104Z tenant=test-f msg="flushing head block failed" err="rename data/test-f/head/01H2Z6QJCQ3STT2E440G9REC4F data/test-f/local/01H2Z6QJCQ3STT2E440G9REC4F: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:40:33.585486326Z tenant=test-e msg="flushing head block failed" err="rename data/test-e/head/01H2Z6SBJGZ2X0Q2S304WMFAF4 data/test-e/local/01H2Z6SBJGZ2X0Q2S304WMFAF4: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:40:35.032895443Z tenant=test-f msg="flushing head block failed" err="rename data/test-f/head/01H2Z6SCZRW16FT3TJWRC8NF77 data/test-f/local/01H2Z6SCZRW16FT3TJWRC8NF77: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:40:36.320883206Z tenant=test-x msg="flushing head block failed" err="rename data/test-x/head/01H2Z6SE808RQNXEJ8KQB28KAN data/test-x/local/01H2Z6SE808RQNXEJ8KQB28KAN: no such file or directory"
level=error caller=phlaredb.go:157 ts=2023-06-15T09:40:46.174610745Z tenant=test-xyz msg="flushing head block failed" err="rename data/test-xyz/head/01H2Z6SQVXGF1RNXSHD2Q8J7KS data/test-xyz/local/01H2Z6SQVXGF1RNXSHD2Q8J7KS: no such file or directory"
```

#### To Reproduce

* Variant: read path
* Query a new and otherwise unused tenant `xyz`
* After `--phlaredb.max-block-duration` we will try to flush the block which will result in the error

* Variant: write path
* Ingest a profile into a tenant `abc`
* After two `--phlaredb.max-block-duration` we will try to flush an empty block which will result in the error

#### Proposed changes

* Do not create a Phlare DB for queries on the read path (instead return empty early) that have no head block or local blocks
* Return the fact that no block was written and prevent the error from being locked

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.