kestra-io / kestra-io/plugin-ovhcloud

[plugin-ovhcloud] Monitoring & Logs — Logs Data Platform and Metrics

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
area/backend area/docs area/plugin
Dominant language
Java
Stars
0
Forks
0
Avg merge
18h 6m
Merged PRs (30d)
1

Description

## Summary

Implement the `monitoring` sub-plugin for `plugin-ovhcloud`, integrating OVHcloud's two observability products: **Logs Data Platform (LDP)** — a managed Graylog/OpenSearch log management service — and **Metrics** — a managed Warp10/OpenTSDB/Prometheus time-series platform. Kestra flows can use this plugin to provision log inputs and streams, manage OpenSearch indexes and aliases, query time-series metrics, and automate alerting and access-token lifecycle for monitoring infrastructure.

## Motivation

Observability pipelines — provisioning log shippers, rotating access tokens, creating alert rules, or exporting metric data for reporting — are today managed by ad hoc scripts or manually through the OVHcloud console. A Kestra plugin lets platform teams encode these as reproducible flows: onboard a new service's logs, create its dashboard, and set up alerts, all in one flow run.

## Context

Part of the OVHcloud plugin EPIC: https://github.com/kestra-io/plugin-ovhcloud/issues/2.
Reference implementation: `plugin-elasticsearch` (OpenSearch index management) and `plugin-core.http` (generic REST queries for Prometheus/Warp10 data-plane). The Metrics product exposes a Prometheus-compatible query endpoint (`https://prometheus..metrics.ovh.net`) — consider a `QueryMetrics` task that wraps PromQL queries for dashboarding flows. The `erlenmeyer` proxy (https://github.com/ovh/erlenmeyer) translates between OpenTSDB, PromQL, InfluxQL, Graphite, and WarpScript on top of Warp10.

## API Reference

- **Official docs (LDP)**: https://eu.api.ovh.com/console/?section=%2Fdbaas%2Flogs&branch=v1
- **Official docs (Metrics)**: https://eu.api.ovh.com/console/?section=%2Fmetrics&branch=v1
- **LDP introduction**: https://help.ovhcloud.com/csm/en-logs-data-platform-introduction
- **Prometheus for Managed Databases**: https://help.ovhcloud.com/csm/en-public-cloud-databases-service-metrics-with-prometheus
- **Authentication**: OVH 3-key HMAC (management plane); LDP stream tokens and Metrics access tokens for data-plane queries
- **Base URL**: `https://eu.api.ovh.com/1.0/`
- **Prometheus data plane**: `https://prometheus..metrics.ovh.net/prometheus/api/v1/`
- **OpenTSDB data plane**: `https://opentsdb..metrics.ovh.net/api/`

## Gradle Dependencies

Add to `build.gradle`:

```groovy
// OVH management plane
implementation 'net.minidev:ovh-java-sdk-core:1.0.17'
implementation 'net.minidev:ovh-java-sdk-dbaaslogs:1.0.17'
implementation 'net.minidev:ovh-java-sdk-dbaastimeseries:1.0.17'

// HTTP client for Prometheus/Warp10 data-plane queries
implementation 'com.squareup.okhttp3:okhttp:4.12.0'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.17.0'
```

> Use the latest stable version available on Maven Central.

## Plugin Structure

- **Repository**: `plugin-ovhcloud`
- **Namespace**: `io.kestra.plugin.ovhcloud`
- **Sub-plugins**: `monitoring.logs`, `monitoring.metrics`

## Suggested Tasks

1. **`monitoring.logs`** — `ListServices`, `GetService`; `ListInputs`, `CreateInput`, `StartInput`, `StopInput`
2. **`monitoring.logs`** — `ListStreams`, `CreateStream`, `AddStreamRule`, `AddStreamAlert`
3. **`monitoring.logs`** — `ListIndexes` (OpenSearch), `CreateIndex`, `ListAliases`, `CreateAlias`
4. **`monitoring.metrics`** — `ListServices`, `CreateToken`, `RevokeToken`, `GetConsumption`
5. **`monitoring.metrics`** — `QueryPrometheus` (PromQL instant/range query against the Prometheus data-plane endpoint)
6. Add polling trigger `monitoring.logs.NewLogEventTrigger` (polls a Graylog stream for new events matching a search query)
7. Write unit + integration tests
8. Add `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.MONITORING)`
9. Add YAML examples and plugin documentation

## YAML Examples

### Example 1 — Provision a new LDP log input and start it

```yaml
id: setup_log_ingestion
namespace: company.team

inputs:
- id: service_name
type: STRING

tasks:
- id: create_input
type: io.kestra.plugin.ovhcloud.monitoring.logs.CreateInput
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
serviceName: "{{ inputs.service_name }}"
description: "Kestra flow log input"
engineType: LOGSTASH
singleInstanceEnabled: false

- id: start_input
type: io.kestra.plugin.ovhcloud.monitoring.logs.StartInput
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
serviceName: "{{ inputs.service_name }}"
inputId: "{{ outputs.create_input.inputId }}"

- id: log
type: io.kestra.plugin.core.log.Log
message: "LDP input {{ outputs.create_input.inputId }} is now running"
```

### Example 2 — Query a Prometheus metric and log the result

```yaml
id: query_metrics_dashboard
namespace: company.team

tasks:
- id: create_metrics_token
type: io.kestra.plugin.ovhcloud.monitoring.metrics.CreateToken
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
serviceName: "{{ secret('METRICS_SERVICE_NAME') }}"
description: "kestra-read-token"
permission: READ

- id: query_cpu
type: io.kestra.plugin.ovhcloud.monitoring.metrics.QueryPrometheus
region: "gra"
token: "{{ outputs.create_metrics_token.value }}"
query: 'avg(cpu_usage_percent{job="web-servers"})'
time: "{{ execution.startDate }}"

- id: log_result
type: io.kestra.plugin.core.log.Log
message: "Average CPU: {{ outputs.query_cpu.result }} %"
```

### Example 3 — React to new log events matching a pattern in a Graylog stream

```yaml
id: on_error_log_event
namespace: company.team

triggers:
- id: watch_error_stream
type: io.kestra.plugin.ovhcloud.monitoring.logs.NewLogEventTrigger
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
serviceName: "{{ secret('LDP_SERVICE_NAME') }}"
streamId: "{{ secret('STREAM_ID') }}"
query: "level:ERROR"
interval: PT5M

tasks:
- id: handle_error
type: io.kestra.plugin.core.log.Log
message: "New error event in LDP: {{ trigger.message }} (source: {{ trigger.source }})"
```

## Acceptance Criteria

- [ ] LDP `CreateInput`, `StartInput`, `StopInput`, `ListInputs` tasks
- [ ] LDP `CreateStream`, `AddStreamRule`, `AddStreamAlert` tasks
- [ ] LDP OpenSearch index and alias management tasks
- [ ] Metrics `CreateToken`, `RevokeToken`, `GetConsumption` tasks
- [ ] Metrics `QueryPrometheus` task (PromQL instant + range query)
- [ ] At least one polling trigger (`NewLogEventTrigger`)
- [ ] All `Property` fields support Kestra expression language
- [ ] Unit + integration tests pass (`./gradlew test`)
- [ ] `package-info.java` with `@PluginSubGroup`
- [ ] Build passes with `./gradlew build`

---

## Repository Setup Checklist

> The repository is shared with other sub-plugins — scaffold once from the EPIC (https://github.com/kestra-io/plugin-ovhcloud/issues/2). Skip if already done.

### 1. Scaffold the repository

Create the repository using the Kestra plugin scaffold tool:
https://github.com/kestra-io/plugins-devtools#kestra-plugin-scaffold

### 2. Add to Sanity check page

Add this plugin to the [Sanity check Notion page](https://www.notion.so/kestra-io/32736907f7b580cbb00dc7c061e624b1?v=32736907f7b58002ac2b000ccc63d8a2).

### 3. Run scoped Terraform apply

Run the following from `infra/terraform/github`:

```bash
terraform apply \
-target='github_repository.repo["plugin-ovhcloud"]' \
-target='github_issue_labels.plugins["plugin-ovhcloud"]' \
-target='github_repository_ruleset.branch["plugin-ovhcloud"]'
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by checking EPIC #2 and whether the shared plugin scaffold already exists. Review the requested dependencies in build.gradle, the referenced plugin implementations, and the package-info.java requirement before separating the logs, metrics, trigger, examples, and documentation work. Done means the listed acceptance criteria pass, including ./gradlew test and ./gradlew build.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, cloud, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.