ISISComputingGroup / ISISComputingGroup/IBEX

Archiver: explore a time-series database

Open
#8,563 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
6
Forks
2
Avg merge
16h 40m
Merged PRs (30d)
2

Description

# Issue Description

As a developer I would like to understand whether a time-series database such as [influxdb](https://www.influxdata.com/products/influxdb/) or `timescaledb` would be more supportable / easier to adapt to our needs than the archiver appliance.

This is exploratory work, where the goal is to form an opinion on whether this would be better/worse than the archive appliance implementation we started doing a PoC of.

**This ticket should be timeboxed**.

---

Some things we want in a new archiver in addition to what we have are:
- Move away from CSS archiver as part of ongoing move away from css tools.
- Support for decimation/downsampling of older data
* So that we can stop manually truncating DBs
* Multi-stage is nice, but a single stage would probably be sufficient in reality.
- Support for configurable retention policies (eventually dropping old data)
* So that we can stop manually truncating DBs
- Support for adding/removing/reconfiguring pvs to archive without restarting entire archiver
- Some way of displaying in _existing_ GUI - this could mean something that it already supported by databrowser e.g. influx, rdb, pbraw, or we'll need to write code to support it.
- "support" (or at least, the archiver doesn't mind too much) if:
* PVs change datatype over time (e.g. blocks may do this if one IOC uses ints and another uses float for the same measurement - and both define the same block)
* Lots of PVs are disconnected (this is something the existing CSS archiver does ok, but the archive appliance struggles with)
- Generally handles pv disconnection/reconnection gracefully and doesn't assume "most" pvs will be connected "most" of the time

### How & Where?

Some items for the reading list:
- https://accelconf.web.cern.ch/icalepcs2023/papers/tupdp108.pdf (from our colleagues over on the ISIS accelerator - we can also just talk to them about what they like/dislike...)
- https://indico.jacow.org/event/80/contributions/5947/contribution.pdf
- https://inspirehep.net/files/0b1eac4f436be0b3e220459d5d4cea9f
- https://accelconf.web.cern.ch/icalepcs2017/papers/thpha032.pdf
- https://github.com/ControlSystemStudio/cs-studio/pull/2246/files
* hypothetically if we wrote our data in that _format_ into influxdb, without necessarily actually using the CSS archiver, the existing databrowser in the GUI would likely work... this is something I think is worth exploring.
* At the very least this is an archive implementation where someone has already used influxdb, so will likely contain useful implementation ideas/inspiration.

# Acceptance Criteria

- [ ] Read the above material and any other related material around archiving into timeseries databases at comparable facilities
- [ ] Write an exploratory proof-of-concept that puts data into a timeseries database (influxdb or timescaledb is suggested, but if an obvious alternative emerges that is also fine)
* Decimation/downsampling - prove it's possible (it should be simple!)
- [ ] Document any pros/cons of the approach
- [ ] Generate tickets to further the implementation

# How to Review
Before making a PR...
- _Provide verbose instructions for the reviewer to test that your changes work and fix the issue_
- _Describe if/how you have implemented testing for this issue_
- _Provide screenshots of the feature to help the reviewer if relevant_

_If not applicable, write "Not applicable"_

[comment]: <> (FILL IN BELOW)
> ...

**To the reviewer: Make sure to [update submodules](https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/Git-workflow#reviewing-work-for-the-submodules-of-epics)!**

Time in recording of planning meeting: 00h15m 2024/11/28

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the listed archiving papers, the CSS archiver context, and the linked ControlSystemStudio pull request, then review the existing databrowser support for influx, RDB, and PBRAW. A timeboxed proof of concept should write data to a time-series database, demonstrate downsampling, document pros and cons, and produce follow-up implementation tickets.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.