ISISComputingGroup / ISISComputingGroup/IBEX

Archiver: explore a time-series database

オープン
#8,563 コメント 4 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
言語のデータがありません
スター
6
フォーク
2
平均マージ
16時間 40分
マージ済み PR(30日)
2

説明

# Issue Description

As a developer I would like to understand whether a time-series database such as [influxdb](https://www.influxdata.com/products/influxdb/) or `timescaledb` would be more supportable / easier to adapt to our needs than the archiver appliance.

This is exploratory work, where the goal is to form an opinion on whether this would be better/worse than the archive appliance implementation we started doing a PoC of.

**This ticket should be timeboxed**.

---

Some things we want in a new archiver in addition to what we have are:
- Move away from CSS archiver as part of ongoing move away from css tools.
- Support for decimation/downsampling of older data
* So that we can stop manually truncating DBs
* Multi-stage is nice, but a single stage would probably be sufficient in reality.
- Support for configurable retention policies (eventually dropping old data)
* So that we can stop manually truncating DBs
- Support for adding/removing/reconfiguring pvs to archive without restarting entire archiver
- Some way of displaying in _existing_ GUI - this could mean something that it already supported by databrowser e.g. influx, rdb, pbraw, or we'll need to write code to support it.
- "support" (or at least, the archiver doesn't mind too much) if:
* PVs change datatype over time (e.g. blocks may do this if one IOC uses ints and another uses float for the same measurement - and both define the same block)
* Lots of PVs are disconnected (this is something the existing CSS archiver does ok, but the archive appliance struggles with)
- Generally handles pv disconnection/reconnection gracefully and doesn't assume "most" pvs will be connected "most" of the time

### How & Where?

Some items for the reading list:
- https://accelconf.web.cern.ch/icalepcs2023/papers/tupdp108.pdf (from our colleagues over on the ISIS accelerator - we can also just talk to them about what they like/dislike...)
- https://indico.jacow.org/event/80/contributions/5947/contribution.pdf
- https://inspirehep.net/files/0b1eac4f436be0b3e220459d5d4cea9f
- https://accelconf.web.cern.ch/icalepcs2017/papers/thpha032.pdf
- https://github.com/ControlSystemStudio/cs-studio/pull/2246/files
* hypothetically if we wrote our data in that _format_ into influxdb, without necessarily actually using the CSS archiver, the existing databrowser in the GUI would likely work... this is something I think is worth exploring.
* At the very least this is an archive implementation where someone has already used influxdb, so will likely contain useful implementation ideas/inspiration.

# Acceptance Criteria

- [ ] Read the above material and any other related material around archiving into timeseries databases at comparable facilities
- [ ] Write an exploratory proof-of-concept that puts data into a timeseries database (influxdb or timescaledb is suggested, but if an obvious alternative emerges that is also fine)
* Decimation/downsampling - prove it's possible (it should be simple!)
- [ ] Document any pros/cons of the approach
- [ ] Generate tickets to further the implementation

# How to Review
Before making a PR...
- _Provide verbose instructions for the reviewer to test that your changes work and fix the issue_
- _Describe if/how you have implemented testing for this issue_
- _Provide screenshots of the feature to help the reviewer if relevant_

_If not applicable, write "Not applicable"_

[comment]: <> (FILL IN BELOW)
> ...

**To the reviewer: Make sure to [update submodules](https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/Git-workflow#reviewing-work-for-the-submodules-of-epics)!**

Time in recording of planning meeting: 00h15m 2024/11/28

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。