ISISComputingGroup / ISISComputingGroup/IBEX
Archiver: explore a time-series database
- Lingua principale
- Nessun dato sulla lingua
- Stelle
- 6
- Fork
- 2
- Merge medio
- 16h 40m
- PR unite (30g)
- 2
Descrizione
# Issue Description
As a developer I would like to understand whether a time-series database such as [influxdb](https://www.influxdata.com/products/influxdb/) or `timescaledb` would be more supportable / easier to adapt to our needs than the archiver appliance.
This is exploratory work, where the goal is to form an opinion on whether this would be better/worse than the archive appliance implementation we started doing a PoC of.
**This ticket should be timeboxed**.
---
Some things we want in a new archiver in addition to what we have are:
- Move away from CSS archiver as part of ongoing move away from css tools.
- Support for decimation/downsampling of older data
* So that we can stop manually truncating DBs
* Multi-stage is nice, but a single stage would probably be sufficient in reality.
- Support for configurable retention policies (eventually dropping old data)
* So that we can stop manually truncating DBs
- Support for adding/removing/reconfiguring pvs to archive without restarting entire archiver
- Some way of displaying in _existing_ GUI - this could mean something that it already supported by databrowser e.g. influx, rdb, pbraw, or we'll need to write code to support it.
- "support" (or at least, the archiver doesn't mind too much) if:
* PVs change datatype over time (e.g. blocks may do this if one IOC uses ints and another uses float for the same measurement - and both define the same block)
* Lots of PVs are disconnected (this is something the existing CSS archiver does ok, but the archive appliance struggles with)
- Generally handles pv disconnection/reconnection gracefully and doesn't assume "most" pvs will be connected "most" of the time
### How & Where?
Some items for the reading list:
- https://accelconf.web.cern.ch/icalepcs2023/papers/tupdp108.pdf (from our colleagues over on the ISIS accelerator - we can also just talk to them about what they like/dislike...)
- https://indico.jacow.org/event/80/contributions/5947/contribution.pdf
- https://inspirehep.net/files/0b1eac4f436be0b3e220459d5d4cea9f
- https://accelconf.web.cern.ch/icalepcs2017/papers/thpha032.pdf
- https://github.com/ControlSystemStudio/cs-studio/pull/2246/files
* hypothetically if we wrote our data in that _format_ into influxdb, without necessarily actually using the CSS archiver, the existing databrowser in the GUI would likely work... this is something I think is worth exploring.
* At the very least this is an archive implementation where someone has already used influxdb, so will likely contain useful implementation ideas/inspiration.
# Acceptance Criteria
- [ ] Read the above material and any other related material around archiving into timeseries databases at comparable facilities
- [ ] Write an exploratory proof-of-concept that puts data into a timeseries database (influxdb or timescaledb is suggested, but if an obvious alternative emerges that is also fine)
* Decimation/downsampling - prove it's possible (it should be simple!)
- [ ] Document any pros/cons of the approach
- [ ] Generate tickets to further the implementation
# How to Review
Before making a PR...
- _Provide verbose instructions for the reviewer to test that your changes work and fix the issue_
- _Describe if/how you have implemented testing for this issue_
- _Provide screenshots of the feature to help the reviewer if relevant_
_If not applicable, write "Not applicable"_
[comment]: <> (FILL IN BELOW)
> ...
**To the reviewer: Make sure to [update submodules](https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/Git-workflow#reviewing-work-for-the-submodules-of-epics)!**
Time in recording of planning meeting: 00h15m 2024/11/28
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.