gchq / gchq/Maestro

Create hooks to enable operation execution monitoring

Open
#2 1 comment 0 reactions 1 assignee Assigned to @GCHQ-9182621 View on GitHub
Dominant language
JavaScript
Stars
4
Forks
3
PR merge metrics
No merged PRs in 30d

Description

It would be great if hooks were available that send out application monitoring messages that can be collected in a central monitoring system. Then information about operation chains that execute across a set of federated services can be gathered and visualised in one place.

So consider an operation chain ([operationA, executor1] -> [[operationB1 -> operationB2], executor2] -> [operationC, executor3]). The federated operation chain handler is going to run through these operations one by one, delegating to the relevant executor for each operation in the chain.

If each of the executors report back information about the operations they are carrying out and the context in which they are doing it (an executor id, the job id of the operation chain, the timestamp of beginning and end of operation, for example), then by aggregating this information we can see the state of the distributed workflow in one place.

In future, this information could be mined to inform users about their operations. It would be possible to estimate the runtime of an operation chain based on previous execution statistics, for example.

The logs could either be sent to an external logging system, or collected as part of the existing job tracker. We could pass the configured monitoring endpoint between systems using the operation context.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.