kestra-io / kestra-io/plugin-scaleway
Scaleway plugin — Object Storage tasks and triggers (S3-compatible)
- Dominant language
- Java
- Stars
- 0
- Forks
- 0
- Avg merge
- 7h 18m
- Merged PRs (30d)
- 2
Description
## Summary
Implement tasks and triggers for [Scaleway Object Storage](https://www.scaleway.com/en/object-storage/) — a fully S3-compatible managed object store available across Scaleway regions. This enables Kestra users to read, write, and manage files in Scaleway buckets directly from their flows without custom HTTP tasks or shell scripts.
## Motivation
Object Storage is the most common integration point between Kestra and cloud providers. Teams using Scaleway today must resort to generic HTTP requests or subprocess calls to transfer files, list objects, or react to bucket events. A native set of tasks and triggers will:
- Allow data pipelines to store outputs (CSVs, Parquet, JSON exports) in Scaleway buckets
- Let orchestration flows trigger on file arrival (polling trigger on new objects)
- Replace error-prone shell-script-based `aws s3 cp` calls with type-safe, documented Kestra tasks
Scaleway Object Storage is S3-compatible, so the implementation should reuse the AWS SDK v2 pattern from `plugin-aws`, configured with a custom S3 endpoint.
## Context
Sub-issue of the Scaleway plugin EPIC. Modeled after `io.kestra.plugin.aws.s3` tasks in `plugin-aws`. The S3-compatible endpoint for Scaleway is `https://s3.{region}.scw.cloud` (e.g., `https://s3.fr-par.scw.cloud`).
## API Reference
- **Official docs**: https://www.scaleway.com/en/developers/api/ (Object Storage section)
- **Authentication**: AWS Signature Version 4 — Scaleway access key as `AWS_ACCESS_KEY_ID`, secret key as `AWS_SECRET_ACCESS_KEY`
- **Base URL pattern**: `https://s3.{region}.scw.cloud` (e.g., `https://s3.fr-par.scw.cloud`)
- **SDK / client library**: AWS SDK v2 `software.amazon.awssdk:s3` configured with endpoint override
## Gradle Dependencies
```groovy
// AWS SDK v2 — S3-compatible Object Storage
implementation "software.amazon.awssdk:s3:2.31.49"
```
> Use the latest stable version available on Maven Central.
## Plugin Structure
- **Repository**: `plugin-scaleway`
- **Namespace**: `io.kestra.plugin.scaleway.storage`
- **Sub-plugins**: `storage`
- **Categories**: `CLOUD`
## Suggested Tasks
1. `AbstractS3` — abstract base class with `accessKey`, `secretKey`, `region` as `Property` fields; builds an `S3Client` with `endpointOverride("https://s3.{region}.scw.cloud")`
2. `CreateBucket` — create a bucket in the specified region
3. `DeleteBucket` — delete a bucket (optionally with force-empty)
4. `ListBuckets` — list all buckets in the project
5. `Upload` — upload a file or Kestra internal storage URI to a bucket
6. `Download` — download an object to Kestra internal storage
7. `Delete` — delete an object by key
8. `List` — list objects in a bucket with optional prefix and pagination
9. `Copy` — copy an object within or across buckets
10. `ObjectStorageTrigger` — polling trigger that fires when new objects appear matching a prefix/regex pattern
## YAML Examples
### Example 1 — Upload a pipeline output to Scaleway Object Storage
```yaml
id: export_to_scaleway_storage
namespace: company.data
tasks:
- id: generate_report
type: io.kestra.plugin.scripts.python.Script
outputFiles:
- report.csv
script: |
with open('report.csv', 'w') as f:
f.write("id,revenue\n1,1000\n2,2500\n")
- id: upload_report
type: io.kestra.plugin.scaleway.storage.Upload
accessKey: "{{ secret('SCW_ACCESS_KEY') }}"
secretKey: "{{ secret('SCW_SECRET_KEY') }}"
region: fr-par
bucket: my-data-lake
key: "reports/{{ execution.startDate | dateFormat('yyyy-MM-dd') }}/report.csv"
from: "{{ outputs.generate_report.outputFiles['report.csv'] }}"
```
### Example 2 — List and log all objects in a bucket with a given prefix
```yaml
id: audit_scaleway_storage
namespace: company.platform
tasks:
- id: list_objects
type: io.kestra.plugin.scaleway.storage.List
accessKey: "{{ secret('SCW_ACCESS_KEY') }}"
secretKey: "{{ secret('SCW_SECRET_KEY') }}"
region: fr-par
bucket: my-data-lake
prefix: reports/
- id: log_count
type: io.kestra.plugin.core.log.Log
message: "Found {{ outputs.list_objects.objects | length }} objects under reports/"
```
### Example 3 — Trigger a flow when a new file lands in a bucket
```yaml
id: process_new_uploads
namespace: company.data
triggers:
- id: on_new_object
type: io.kestra.plugin.scaleway.storage.ObjectStorageTrigger
accessKey: "{{ secret('SCW_ACCESS_KEY') }}"
secretKey: "{{ secret('SCW_SECRET_KEY') }}"
region: fr-par
bucket: my-data-lake
prefix: incoming/
interval: PT2M
tasks:
- id: download_file
type: io.kestra.plugin.scaleway.storage.Download
accessKey: "{{ secret('SCW_ACCESS_KEY') }}"
secretKey: "{{ secret('SCW_SECRET_KEY') }}"
region: fr-par
bucket: my-data-lake
key: "{{ trigger.key }}"
- id: process
type: io.kestra.plugin.core.log.Log
message: "Processing {{ trigger.key }} ({{ trigger.size }} bytes)"
```
## Acceptance Criteria
- [ ] `AbstractS3` base class with `accessKey`, `secretKey`, `region` as `Property`, builds `S3Client` with Scaleway endpoint override
- [ ] `CreateBucket`, `DeleteBucket`, `ListBuckets` tasks implemented
- [ ] `Upload`, `Download`, `Delete`, `List`, `Copy` tasks implemented
- [ ] `ObjectStorageTrigger` polling trigger implemented
- [ ] All `Property` fields support Kestra expression language (template rendering)
- [ ] Unit tests with mocked S3 responses; integration tests against live Scaleway (skipped if credentials absent)
- [ ] `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.CLOUD)`
- [ ] Build passes with `./gradlew build`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the AWS S3 task patterns in plugin-aws and the requested package-info.java entry, then inspect the Gradle configuration for the SDK dependency. Implement the listed bucket and object tasks plus the polling trigger, add mocked and credential-gated integration tests, and run ./gradlew build; completion is all acceptance criteria passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100