kestra-io / kestra-io/plugin-scaleway

Scaleway plugin — EPIC

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
area/plugin
Dominant language
Java
Stars
0
Forks
0
Avg merge
7h 18m
Merged PRs (30d)
2

Description

## Summary

The Scaleway plugin initiative brings first-class support for [Scaleway](https://www.scaleway.com/en/) — a leading European cloud provider — into the Kestra ecosystem. Scaleway offers a broad portfolio of cloud services (compute, serverless, object storage, managed databases, queues, secret management, and more) widely adopted by European data and engineering teams. This EPIC tracks the full lifecycle of `plugin-scaleway`, from initial scaffold to production-ready tasks and triggers across all service groups.

## Motivation

Scaleway has strong adoption across European organizations looking for GDPR-compliant cloud infrastructure. Today, teams using Scaleway alongside Kestra must rely on generic HTTP tasks, shell scripts, or custom Python code to interact with Scaleway APIs. A native plugin removes this friction and unlocks:

- Seamless orchestration of Scaleway compute, serverless, and storage resources from Kestra flows
- Event-driven workflows triggered by Scaleway queue messages or infrastructure events
- Secure runtime secret injection from Scaleway Secret Manager
- European data-residency-aware pipelines without leaving the Kestra UI

## API Reference

- **Official docs**: https://www.scaleway.com/en/developers/api/
- **Authentication**: All REST requests use `X-Auth-Token: `. API keys are created in the Scaleway Console or via the IAM API. S3/SQS/SNS endpoints use AWS Signature Version 4 with the same access key + secret key pair.
- Key creation docs: https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/
- **Base URL pattern**: `https://api.scaleway.com/{product}/{version}/zones/{zone}/` (zonal) or `/regions/{region}/` (regional)
- **SDK**: No official Java SDK. Use AWS SDK v2 for S3/SQS/SNS surfaces; Apache HttpClient 5 + Jackson for all other REST services.

## Shared Architecture — `ScalewayConnection`

All tasks and triggers must extend a shared abstract base class `ScalewayConnection` (or per-surface variants) that centralises credentials and client construction:

```
io.kestra.plugin.scaleway.AbstractScalewayConnection
├── Property secretKey // X-Auth-Token for REST APIs
├── Property accessKey // Used for S3/SQS/SNS SigV4 auth
├── Property region // e.g. "fr-par", "nl-ams"
├── Property zone // e.g. "fr-par-1" (zonal resources only)
├── S3Client buildS3Client() // endpoint: https://s3.{region}.scw.cloud
├── SqsClient buildSqsClient() // endpoint: https://sqs.mnq.{region}.scaleway.com
├── SnsClient buildSnsClient() // endpoint: https://sns.mnq.{region}.scaleway.com
└── CloseableHttpClient buildHttpClient() // X-Auth-Token header pre-configured
```

## Gradle Dependencies

All sub-modules share these dependencies (add to root `build.gradle`):

```groovy
// AWS SDK v2 — S3-compatible Object Storage
implementation "software.amazon.awssdk:s3:2.31.49"

// AWS SDK v2 — SQS-compatible Queues
implementation "software.amazon.awssdk:sqs:2.31.49"

// AWS SDK v2 — SNS-compatible Topics
implementation "software.amazon.awssdk:sns:2.31.49"

// HTTP client for Scaleway REST API (Instances, Functions, Containers, RDB, Secrets, Registry)
implementation "org.apache.httpcomponents.client5:httpclient5:5.3.1"

// JSON serialization
implementation "com.fasterxml.jackson.core:jackson-databind:2.17.0"
```

> Use the latest stable versions available on Maven Central.

## Plugin Structure

- **Repository**: `plugin-scaleway`
- **Namespace**: `io.kestra.plugin.scaleway`
- **Sub-packages**:

| Sub-package | Namespace | Service |
|---|---|---|
| `storage` | `io.kestra.plugin.scaleway.storage` | Object Storage (S3-compatible) |
| `instance` | `io.kestra.plugin.scaleway.instance` | Compute Instances |
| `network` | `io.kestra.plugin.scaleway.network` | VPC, Private Networks, Flexible IPs, Security Groups |
| `functions` | `io.kestra.plugin.scaleway.functions` | Serverless Functions |
| `containers` | `io.kestra.plugin.scaleway.containers` | Serverless Containers |
| `rdb` | `io.kestra.plugin.scaleway.rdb` | Managed Database (PostgreSQL / MySQL) |
| `secrets` | `io.kestra.plugin.scaleway.secrets` | Secret Manager |
| `registry` | `io.kestra.plugin.scaleway.registry` | Container Registry |
| `queues` | `io.kestra.plugin.scaleway.queues` | Queues (SQS-compatible) |
| `topics` | `io.kestra.plugin.scaleway.topics` | Topics and Events (SNS-compatible) |
| `kubernetes` | `io.kestra.plugin.scaleway.kubernetes` | Kubernetes Kapsule (managed Kubernetes) — see #23 |
| `genai` | `io.kestra.plugin.scaleway.genai` | Generative APIs (OpenAI-compatible LLM inference) — see #24 |

Each sub-package has its own `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.CLOUD)`, except `genai` which uses `PluginCategory.AI`.

## Sub-issues

## Acceptance Criteria

- [ ] `AbstractScalewayConnection` base class with `secretKey`, `accessKey`, `region`, `zone` and client factory methods
- [ ] All sub-issues listed above are implemented, reviewed, and merged
- [ ] Each sub-package has `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.CLOUD)`
- [ ] `metadata/index.yaml` and plugin icon SVG present at the repository root
- [ ] All tasks and triggers support Kestra expression language (template rendering)
- [ ] Integration documented in the Kestra docs

## Repository Setup Checklist

### 1. Add to Sanity check page

Add this plugin to the [Sanity check Notion page](https://www.notion.so/kestra-io/32736907f7b580cbb00dc7c061e624b1?v=32736907f7b58002ac2b000ccc63d8a2).

### 2. Run scoped Terraform apply

Run the following from `infra/terraform/github`:

```bash
terraform apply \
-target='github_repository.repo["plugin-scaleway"]' \
-target='github_issue_labels.plugins["plugin-scaleway"]' \
-target='github_repository_ruleset.branch["plugin-scaleway"]'
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the root build.gradle dependencies, the proposed io.kestra.plugin.scaleway package structure, and the acceptance criteria. Check the linked service sub-issues, then verify the shared connection design, package-info.java files, metadata/index.yaml, icon, documentation, and Terraform setup before treating the epic as complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, terraform
Domain
backend, cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.