Joystream / Joystream/joystream

Lead infrastructure (error) monitoring

Open
#4,452 0 comments 0 reactions 0 assignees View on GitHub
argus colossus
Dominant language
Rust
Stars
1.4k
Forks
116
PR merge metrics
No merged PRs in 30d

Description

# Background

Currently, there is no way for leads, whether storage or distributor, to automatically learn about changes in the error frequency of various actors and services in the overall infrastructure. The closest tool to this is a ES powered hosted service operated by a senior community member which storage provider operates, but it is apparently not very well used, and requires manual intervention for providers to report to, or to update in case of failure of that endpoint.

# Proposal

Create a new lead infrastructure service which monitors errors, satisyfing the following requirements

- `Argus`, `Colossus` and `Orion(v2)` all report content related delivery related requests to this endpoint. In particular `Orion` will be reporting on behalf of errors experienced by clients on that gateway. We here are interested in all interactions, so
- `Orion` user to `Colossus` uploads
- `Orion` user to `Argus` downloads
- `Colossus` from `Colossus` downloads
- `Argus` from `Colossus` downloads
- There is on-chain metadata which is publishable by lead, which sets where to resolve this endpoint at any given time. All nodes follow this, unless explicitly configured not to. API should report what endpoint the node is currently reporting to.
- All error claims have to be done with identifier for reporter, along with attestation using signature for some appropriate role key of actor. Should be one of the metaprotocol level membership keys, so does not need to have funds.
- All errors much use accurate on-chain identifiers of all data and actors being referenced, and time/block,so as to later make attribution easy, and also cross referencing across errors.
- It should be possible to trigger hooks that send notifications to key actors over transports like email, sms or Discord, when certain global or actors specific error frequencies pass some adjustable thresholds.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.