kestra-io / kestra-io/plugin-ovhcloud
[plugin-ovhcloud] Monitoring & Logs — Logs Data Platform and Metrics
- Dominant language
- Java
- Stars
- 0
- Forks
- 0
- Avg merge
- 18h 6m
- Merged PRs (30d)
- 1
Description
## Summary
Implement the `monitoring` sub-plugin for `plugin-ovhcloud`, integrating OVHcloud's two observability products: **Logs Data Platform (LDP)** — a managed Graylog/OpenSearch log management service — and **Metrics** — a managed Warp10/OpenTSDB/Prometheus time-series platform. Kestra flows can use this plugin to provision log inputs and streams, manage OpenSearch indexes and aliases, query time-series metrics, and automate alerting and access-token lifecycle for monitoring infrastructure.
## Motivation
Observability pipelines — provisioning log shippers, rotating access tokens, creating alert rules, or exporting metric data for reporting — are today managed by ad hoc scripts or manually through the OVHcloud console. A Kestra plugin lets platform teams encode these as reproducible flows: onboard a new service's logs, create its dashboard, and set up alerts, all in one flow run.
## Context
Part of the OVHcloud plugin EPIC: https://github.com/kestra-io/plugin-ovhcloud/issues/2.
Reference implementation: `plugin-elasticsearch` (OpenSearch index management) and `plugin-core.http` (generic REST queries for Prometheus/Warp10 data-plane). The Metrics product exposes a Prometheus-compatible query endpoint (`https://prometheus..metrics.ovh.net`) — consider a `QueryMetrics` task that wraps PromQL queries for dashboarding flows. The `erlenmeyer` proxy (https://github.com/ovh/erlenmeyer) translates between OpenTSDB, PromQL, InfluxQL, Graphite, and WarpScript on top of Warp10.
## API Reference
- **Official docs (LDP)**: https://eu.api.ovh.com/console/?section=%2Fdbaas%2Flogs&branch=v1
- **Official docs (Metrics)**: https://eu.api.ovh.com/console/?section=%2Fmetrics&branch=v1
- **LDP introduction**: https://help.ovhcloud.com/csm/en-logs-data-platform-introduction
- **Prometheus for Managed Databases**: https://help.ovhcloud.com/csm/en-public-cloud-databases-service-metrics-with-prometheus
- **Authentication**: OVH 3-key HMAC (management plane); LDP stream tokens and Metrics access tokens for data-plane queries
- **Base URL**: `https://eu.api.ovh.com/1.0/`
- **Prometheus data plane**: `https://prometheus..metrics.ovh.net/prometheus/api/v1/`
- **OpenTSDB data plane**: `https://opentsdb..metrics.ovh.net/api/`
## Gradle Dependencies
Add to `build.gradle`:
```groovy
// OVH management plane
implementation 'net.minidev:ovh-java-sdk-core:1.0.17'
implementation 'net.minidev:ovh-java-sdk-dbaaslogs:1.0.17'
implementation 'net.minidev:ovh-java-sdk-dbaastimeseries:1.0.17'
// HTTP client for Prometheus/Warp10 data-plane queries
implementation 'com.squareup.okhttp3:okhttp:4.12.0'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.17.0'
```
> Use the latest stable version available on Maven Central.
## Plugin Structure
- **Repository**: `plugin-ovhcloud`
- **Namespace**: `io.kestra.plugin.ovhcloud`
- **Sub-plugins**: `monitoring.logs`, `monitoring.metrics`
## Suggested Tasks
1. **`monitoring.logs`** — `ListServices`, `GetService`; `ListInputs`, `CreateInput`, `StartInput`, `StopInput`
2. **`monitoring.logs`** — `ListStreams`, `CreateStream`, `AddStreamRule`, `AddStreamAlert`
3. **`monitoring.logs`** — `ListIndexes` (OpenSearch), `CreateIndex`, `ListAliases`, `CreateAlias`
4. **`monitoring.metrics`** — `ListServices`, `CreateToken`, `RevokeToken`, `GetConsumption`
5. **`monitoring.metrics`** — `QueryPrometheus` (PromQL instant/range query against the Prometheus data-plane endpoint)
6. Add polling trigger `monitoring.logs.NewLogEventTrigger` (polls a Graylog stream for new events matching a search query)
7. Write unit + integration tests
8. Add `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.MONITORING)`
9. Add YAML examples and plugin documentation
## YAML Examples
### Example 1 — Provision a new LDP log input and start it
```yaml
id: setup_log_ingestion
namespace: company.team
inputs:
- id: service_name
type: STRING
tasks:
- id: create_input
type: io.kestra.plugin.ovhcloud.monitoring.logs.CreateInput
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
serviceName: "{{ inputs.service_name }}"
description: "Kestra flow log input"
engineType: LOGSTASH
singleInstanceEnabled: false
- id: start_input
type: io.kestra.plugin.ovhcloud.monitoring.logs.StartInput
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
serviceName: "{{ inputs.service_name }}"
inputId: "{{ outputs.create_input.inputId }}"
- id: log
type: io.kestra.plugin.core.log.Log
message: "LDP input {{ outputs.create_input.inputId }} is now running"
```
### Example 2 — Query a Prometheus metric and log the result
```yaml
id: query_metrics_dashboard
namespace: company.team
tasks:
- id: create_metrics_token
type: io.kestra.plugin.ovhcloud.monitoring.metrics.CreateToken
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
serviceName: "{{ secret('METRICS_SERVICE_NAME') }}"
description: "kestra-read-token"
permission: READ
- id: query_cpu
type: io.kestra.plugin.ovhcloud.monitoring.metrics.QueryPrometheus
region: "gra"
token: "{{ outputs.create_metrics_token.value }}"
query: 'avg(cpu_usage_percent{job="web-servers"})'
time: "{{ execution.startDate }}"
- id: log_result
type: io.kestra.plugin.core.log.Log
message: "Average CPU: {{ outputs.query_cpu.result }} %"
```
### Example 3 — React to new log events matching a pattern in a Graylog stream
```yaml
id: on_error_log_event
namespace: company.team
triggers:
- id: watch_error_stream
type: io.kestra.plugin.ovhcloud.monitoring.logs.NewLogEventTrigger
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
serviceName: "{{ secret('LDP_SERVICE_NAME') }}"
streamId: "{{ secret('STREAM_ID') }}"
query: "level:ERROR"
interval: PT5M
tasks:
- id: handle_error
type: io.kestra.plugin.core.log.Log
message: "New error event in LDP: {{ trigger.message }} (source: {{ trigger.source }})"
```
## Acceptance Criteria
- [ ] LDP `CreateInput`, `StartInput`, `StopInput`, `ListInputs` tasks
- [ ] LDP `CreateStream`, `AddStreamRule`, `AddStreamAlert` tasks
- [ ] LDP OpenSearch index and alias management tasks
- [ ] Metrics `CreateToken`, `RevokeToken`, `GetConsumption` tasks
- [ ] Metrics `QueryPrometheus` task (PromQL instant + range query)
- [ ] At least one polling trigger (`NewLogEventTrigger`)
- [ ] All `Property` fields support Kestra expression language
- [ ] Unit + integration tests pass (`./gradlew test`)
- [ ] `package-info.java` with `@PluginSubGroup`
- [ ] Build passes with `./gradlew build`
---
## Repository Setup Checklist
> The repository is shared with other sub-plugins — scaffold once from the EPIC (https://github.com/kestra-io/plugin-ovhcloud/issues/2). Skip if already done.
### 1. Scaffold the repository
Create the repository using the Kestra plugin scaffold tool:
https://github.com/kestra-io/plugins-devtools#kestra-plugin-scaffold
### 2. Add to Sanity check page
Add this plugin to the [Sanity check Notion page](https://www.notion.so/kestra-io/32736907f7b580cbb00dc7c061e624b1?v=32736907f7b58002ac2b000ccc63d8a2).
### 3. Run scoped Terraform apply
Run the following from `infra/terraform/github`:
```bash
terraform apply \
-target='github_repository.repo["plugin-ovhcloud"]' \
-target='github_issue_labels.plugins["plugin-ovhcloud"]' \
-target='github_repository_ruleset.branch["plugin-ovhcloud"]'
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by checking EPIC #2 and whether the shared plugin scaffold already exists. Review the requested dependencies in build.gradle, the referenced plugin implementations, and the package-info.java requirement before separating the logs, metrics, trigger, examples, and documentation work. Done means the listed acceptance criteria pass, including ./gradlew test and ./gradlew build.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, cloud, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100