kestra-io / kestra-io/plugin-ovhcloud

[plugin-ovhcloud] Storage — Object Storage, Cold Archive, Block Storage, and Managed Databases

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
area/plugin
Dominant language
Java
Stars
0
Forks
0
Avg merge
18h 6m
Merged PRs (30d)
1

Description

## Summary

Implement the `storage` sub-plugin for `plugin-ovhcloud`, covering OVHcloud's full storage portfolio: S3-compatible Object Storage (with per-region endpoints), Cold Archive (tape-backed glacier storage), Block Storage volumes with snapshot support, and Managed Databases (PostgreSQL, MySQL, MongoDB, Redis, Kafka, OpenSearch, and more). This plugin lets data engineering teams move files, manage data tiers, provision databases, and automate cold archiving without leaving a Kestra flow.

## Motivation

Data pipelines running on OVHcloud typically handle three storage tiers — hot (object storage), warm (block), and cold (tape archive) — plus managed databases for state. Managing these manually or via shell scripts is error-prone and hard to audit. A native Kestra plugin makes storage lifecycle automation declarative, observable, and composable with the rest of the data stack.

## Context

Part of the OVHcloud plugin EPIC: https://github.com/kestra-io/plugin-ovhcloud/issues/2.
Reference implementation: `plugin-aws` (S3 tasks + RDS) for structure. Object Storage and Cold Archive share the same S3-compatible API; use AWS SDK v2 with a custom endpoint override. Block Storage and Managed Databases use the OVH REST API with 3-key HMAC auth.

## API Reference

- **Official docs (S3)**: https://help.ovhcloud.com/csm/en-public-cloud-storage-s3-getting-started-object-storage
- **Official docs (Cold Archive)**: https://help.ovhcloud.com/csm/en-public-cloud-storage-cold-archive-getting-started
- **Official docs (Databases)**: https://help.ovhcloud.com/csm/en-public-cloud-databases-order-api
- **Authentication**: AWS Signature V4 for Object Storage + Cold Archive (separate S3 credentials from OVH API keys); OVH 3-key HMAC for Block Storage and Managed Databases
- **S3 endpoint pattern**: `https://s3..io.cloud.ovh.net` (e.g. `https://s3.gra.io.cloud.ovh.net`)
- **OVH API base URL**: `https://eu.api.ovh.com/1.0/`

## Gradle Dependencies

Add to `build.gradle`:

```groovy
// AWS SDK v2 for S3-compatible Object Storage and Cold Archive
implementation platform('software.amazon.awssdk:bom:2.25.70')
implementation 'software.amazon.awssdk:s3'
implementation 'software.amazon.awssdk:s3-transfer-manager'

// OVH management plane for Block Storage and Managed Databases
implementation 'net.minidev:ovh-java-sdk-core:1.0.17'
implementation 'net.minidev:ovh-java-sdk-cloud:1.0.17'
```

> Use the latest stable version available on Maven Central.

## Plugin Structure

- **Repository**: `plugin-ovhcloud`
- **Namespace**: `io.kestra.plugin.ovhcloud`
- **Sub-plugins**: `storage.objectstorage`, `storage.coldarchive`, `storage.block`, `storage.database`

## Suggested Tasks

1. **`storage.objectstorage`** — `CreateBucket`, `Upload`, `Download`, `Delete`, `List` (mirrors `plugin-aws` S3 tasks, custom endpoint)
2. **`storage.coldarchive`** — `Archive` (PUT with storage class GLACIER), `Restore` (POST restore request), `GetRestoreStatus`
3. **`storage.block`** — `ListVolumes`, `CreateVolume`, `DeleteVolume`, `AttachVolume`, `DetachVolume`, `CreateSnapshot`
4. **`storage.database`** — `ListClusters`, `CreateCluster`, `DeleteCluster`, `GetConnectionDetails`; supports all OVHcloud-managed engines
5. Add polling trigger `storage.objectstorage.NewObjectTrigger` (watches for new objects in a bucket prefix)
6. Write unit + integration tests
7. Add `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.STORAGE)`
8. Add `metadata/index.yaml` and plugin icon SVG
9. Add YAML examples and plugin documentation

## YAML Examples

### Example 1 — Upload a file to OVHcloud Object Storage

```yaml
id: upload_to_ovh_s3
namespace: company.team

inputs:
- id: bucket
type: STRING
- id: region
type: STRING
defaults: gra

tasks:
- id: upload_file
type: io.kestra.plugin.ovhcloud.storage.objectstorage.Upload
accessKey: "{{ secret('OVH_S3_ACCESS_KEY') }}"
secretKey: "{{ secret('OVH_S3_SECRET_KEY') }}"
region: "{{ inputs.region }}"
bucket: "{{ inputs.bucket }}"
key: "data/{{ execution.id }}/output.csv"
from: "{{ outputs.previous_task.uri }}"
```

### Example 2 — Archive an object to Cold Archive (tape) and restore it

```yaml
id: cold_archive_workflow
namespace: company.team

tasks:
- id: archive
type: io.kestra.plugin.ovhcloud.storage.coldarchive.Archive
accessKey: "{{ secret('OVH_S3_ACCESS_KEY') }}"
secretKey: "{{ secret('OVH_S3_SECRET_KEY') }}"
region: "gra"
bucket: "my-archive-bucket"
key: "archive/2024/annual-report.parquet"
from: "{{ outputs.generate_report.uri }}"

- id: wait_and_restore
type: io.kestra.plugin.ovhcloud.storage.coldarchive.Restore
accessKey: "{{ secret('OVH_S3_ACCESS_KEY') }}"
secretKey: "{{ secret('OVH_S3_SECRET_KEY') }}"
region: "gra"
bucket: "my-archive-bucket"
key: "archive/2024/annual-report.parquet"
restoreDays: 7
```

### Example 3 — React to new objects arriving in a bucket prefix

```yaml
id: process_incoming_files
namespace: company.team

triggers:
- id: on_new_file
type: io.kestra.plugin.ovhcloud.storage.objectstorage.NewObjectTrigger
accessKey: "{{ secret('OVH_S3_ACCESS_KEY') }}"
secretKey: "{{ secret('OVH_S3_SECRET_KEY') }}"
region: "gra"
bucket: "my-input-bucket"
prefix: "incoming/"
interval: PT5M

tasks:
- id: process
type: io.kestra.plugin.core.log.Log
message: "New file: {{ trigger.objectKey }} ({{ trigger.size }} bytes)"
```

## Acceptance Criteria

- [ ] S3-compatible `Upload`, `Download`, `Delete`, `List`, `CreateBucket` tasks with custom OVHcloud endpoint
- [ ] Cold Archive `Archive`, `Restore`, `GetRestoreStatus` tasks (storage class GLACIER)
- [ ] Block Storage `Create`, `Delete`, `Attach`, `Detach`, `CreateSnapshot` tasks
- [ ] Managed Databases `Create`, `Delete`, `GetConnectionDetails` tasks (multi-engine)
- [ ] At least one polling trigger (`NewObjectTrigger`)
- [ ] All `Property` fields support Kestra expression language
- [ ] Unit + integration tests pass (`./gradlew test`)
- [ ] `package-info.java` with `@PluginSubGroup`
- [ ] Build passes with `./gradlew build`

---

## Repository Setup Checklist

> The repository is shared with other sub-plugins — scaffold once from the EPIC (https://github.com/kestra-io/plugin-ovhcloud/issues/2). Skip if already done.

### 1. Scaffold the repository

Create the repository using the Kestra plugin scaffold tool:
https://github.com/kestra-io/plugins-devtools#kestra-plugin-scaffold

### 2. Add to Sanity check page

Add this plugin to the [Sanity check Notion page](https://www.notion.so/kestra-io/32736907f7b580cbb00dc7c061e624b1?v=32736907f7b58002ac2b000ccc63d8a2).

### 3. Run scoped Terraform apply

Run the following from `infra/terraform/github`:

```bash
terraform apply \
-target='github_repository.repo["plugin-ovhcloud"]' \
-target='github_issue_labels.plugins["plugin-ovhcloud"]' \
-target='github_repository_ruleset.branch["plugin-ovhcloud"]'
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the plugin-aws reference implementation and the proposed build.gradle dependencies, then scaffold the shared plugin repository if needed. Implement the listed storage tasks and trigger, add package-info.java, metadata/index.yaml, documentation, examples, and tests; verify completion with ./gradlew test and ./gradlew build against the acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, kafka, mongodb, mysql, postgresql, redis, terraform
Domain
api, build-system, cloud, data-engineering, database, documentation, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.