kestra-io / kestra-io/plugin-ovhcloud
[plugin-ovhcloud] AI & ML — Training jobs, Deploy apps, and Notebooks
- Dominant language
- Java
- Stars
- 0
- Forks
- 0
- Avg merge
- 18h 6m
- Merged PRs (30d)
- 1
Description
## Summary
Implement the `ai` sub-plugin for `plugin-ovhcloud`, enabling Kestra flows to orchestrate the full OVHcloud AI lifecycle: submit and monitor AI Training jobs (GPU-backed distributed training), deploy and scale AI Deploy apps (model serving endpoints), and manage AI Notebooks. This plugin bridges ML engineering workflows — training, packaging, deploying — with Kestra's orchestration and scheduling capabilities, making MLOps pipelines on OVHcloud fully reproducible.
## Motivation
ML practitioners on OVHcloud today submit training jobs through the `ovhai` CLI or Python SDK and then manually watch for completion before triggering downstream steps (evaluation, packaging, deployment). A native Kestra plugin makes these steps event-driven and composable: train on schedule, wait for job completion, evaluate metrics, deploy if thresholds pass — all in a single flow.
## Context
Part of the OVHcloud plugin EPIC: https://github.com/kestra-io/plugin-ovhcloud/issues/2.
Reference implementation: `plugin-gcp` (Vertex AI jobs) and `plugin-aws` (SageMaker training) for task structure. The OVH AI API sits under `/cloud/project/{serviceName}/ai/*` and uses OVH 3-key HMAC for management; AI tokens (provisioned via the same API) are used for data-plane calls. The Python `ovhai-python-sdk` (https://github.com/ovh/ovhai-python-sdk) is a useful API reference; Kestra tasks will call the REST API directly with `ovh-java-sdk-core` for signing.
## API Reference
- **Official docs**: https://eu.api.ovh.com/console/?section=%2Fcloud&branch=v1 (filter: `/ai`)
- **AI Training guide**: https://help.ovhcloud.com/csm/en-public-cloud-ai-training-submit-job
- **ovhai Python SDK reference**: https://github.com/ovh/ovhai-python-sdk
- **Authentication**: OVH 3-key HMAC for management API; AI bearer token (provisioned via `POST /cloud/project/{serviceName}/ai/token`) for direct `ovhai` data-plane calls
- **Base URL**: `https://eu.api.ovh.com/1.0/cloud/project/{serviceName}/ai/`
## Gradle Dependencies
Add to `build.gradle`:
```groovy
// OVH management plane — HMAC signing (no typed AI module exists in ovh-java-sdk)
implementation 'net.minidev:ovh-java-sdk-core:1.0.17'
// HTTP client for REST calls to AI endpoints
implementation 'com.squareup.okhttp3:okhttp:4.12.0'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.17.0'
```
> Use the latest stable version available on Maven Central.
## Plugin Structure
- **Repository**: `plugin-ovhcloud`
- **Namespace**: `io.kestra.plugin.ovhcloud`
- **Sub-plugins**: `ai.training`, `ai.deploy`, `ai.notebook`
## Suggested Tasks
1. Implement AI token management helpers (create token, manage token lifecycle)
2. **`ai.training`** — `SubmitJob`, `GetJob`, `CancelJob`, `GetJobLogs`; `WaitForJobCompletion` (blocking poll)
3. **`ai.deploy`** — `CreateApp`, `GetApp`, `UpdateApp` (scaling), `DeleteApp`
4. **`ai.notebook`** — `CreateNotebook`, `StartNotebook`, `StopNotebook`, `DeleteNotebook`
5. **`ai.training`** — `ListJobs` with filter by status/label
6. Add polling trigger `ai.training.JobCompletedTrigger` (fires when a training job reaches DONE or FAILED)
7. Write unit + integration tests
8. Add `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.AI)`
9. Add YAML examples and plugin documentation
## YAML Examples
### Example 1 — Submit an AI Training job and wait for completion
```yaml
id: train_model_on_ovhcloud
namespace: company.team
inputs:
- id: project_id
type: STRING
- id: image
type: STRING
defaults: "pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime"
tasks:
- id: submit_job
type: io.kestra.plugin.ovhcloud.ai.training.SubmitJob
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
projectId: "{{ inputs.project_id }}"
image: "{{ inputs.image }}"
command: ["python", "train.py", "--epochs", "50"]
resources:
gpu: 1
gpuModel: "Tesla-V100S"
volumes:
- mountPath: "/data"
objectStorageRegion: "GRA"
container: "training-data"
permission: RO
name: "kestra-training-{{ execution.id }}"
- id: wait_for_completion
type: io.kestra.plugin.ovhcloud.ai.training.WaitForJobCompletion
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
projectId: "{{ inputs.project_id }}"
jobId: "{{ outputs.submit_job.jobId }}"
pollInterval: PT30S
timeout: PT4H
- id: log_result
type: io.kestra.plugin.core.log.Log
message: "Training job finished with status: {{ outputs.wait_for_completion.status }}"
```
### Example 2 — Deploy a model as a serving endpoint
```yaml
id: deploy_model_endpoint
namespace: company.team
tasks:
- id: create_app
type: io.kestra.plugin.ovhcloud.ai.deploy.CreateApp
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
projectId: "{{ secret('OVH_PROJECT_ID') }}"
image: "{{ secret('MODEL_SERVING_IMAGE') }}"
name: "fraud-detector-v2"
resources:
cpu: 4
memory: 8
scalingStrategy:
autoscaling:
averageCpuUsageTarget: 75
minReplicas: 1
maxReplicas: 5
- id: log_endpoint
type: io.kestra.plugin.core.log.Log
message: "App deployed at {{ outputs.create_app.url }}"
```
### Example 3 — React when a training job completes
```yaml
id: on_training_complete
namespace: company.team
triggers:
- id: watch_training_job
type: io.kestra.plugin.ovhcloud.ai.training.JobCompletedTrigger
endpoint: "{{ secret('OVH_ENDPOINT') }}"
applicationKey: "{{ secret('OVH_APP_KEY') }}"
applicationSecret: "{{ secret('OVH_APP_SECRET') }}"
consumerKey: "{{ secret('OVH_CONSUMER_KEY') }}"
projectId: "{{ secret('OVH_PROJECT_ID') }}"
jobNamePrefix: "kestra-training-"
interval: PT2M
tasks:
- id: evaluate
type: io.kestra.plugin.core.log.Log
message: "Job {{ trigger.jobId }} completed with status {{ trigger.status }} — exit code {{ trigger.exitCode }}"
```
## Acceptance Criteria
- [ ] AI token management utilities implemented
- [ ] Training `SubmitJob`, `GetJob`, `CancelJob`, `GetJobLogs`, `WaitForJobCompletion` tasks
- [ ] Deploy `CreateApp`, `GetApp`, `UpdateApp`, `DeleteApp` tasks
- [ ] Notebook `CreateNotebook`, `StartNotebook`, `StopNotebook`, `DeleteNotebook` tasks
- [ ] At least one polling trigger (`JobCompletedTrigger`)
- [ ] All `Property` fields support Kestra expression language
- [ ] Unit + integration tests pass (`./gradlew test`)
- [ ] `package-info.java` with `@PluginSubGroup`
- [ ] Build passes with `./gradlew build`
---
## Repository Setup Checklist
> The repository is shared with other sub-plugins — scaffold once from the EPIC (https://github.com/kestra-io/plugin-ovhcloud/issues/2). Skip if already done.
### 1. Scaffold the repository
Create the repository using the Kestra plugin scaffold tool:
https://github.com/kestra-io/plugins-devtools#kestra-plugin-scaffold
### 2. Add to Sanity check page
Add this plugin to the [Sanity check Notion page](https://www.notion.so/kestra-io/32736907f7b580cbb00dc7c061e624b1?v=32736907f7b58002ac2b000ccc63d8a2).
### 3. Run scoped Terraform apply
Run the following from `infra/terraform/github`:
```bash
terraform apply \
-target='github_repository.repo["plugin-ovhcloud"]' \
-target='github_issue_labels.plugins["plugin-ovhcloud"]' \
-target='github_repository_ruleset.branch["plugin-ovhcloud"]'
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by scaffolding the shared plugin repository, then read the plugin-gcp and plugin-aws task structures, the OVHcloud AI API reference, and build.gradle. Implement the listed training, deployment, notebook, token, trigger, documentation, and package-info.java work; verify with ./gradlew test and ./gradlew build.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ai, backend-api-design, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100