kestra-io / kestra-io/plugin-oci

[Plugin] OCI — Object Storage (Buckets & Objects)

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
area/plugin
Dominant language
Java
Stars
0
Forks
0
Avg merge
18h 3m
Merged PRs (30d)
1

Description

## Summary

The OCI Object Storage sub-plugin for `plugin-oci` lets Kestra flows create and manage buckets, upload and download objects, list bucket contents, and delete objects — all natively within a flow. It turns OCI Object Storage into a first-class data handoff layer between tasks, complementing the existing S3 and GCS plugins for teams standardized on Oracle Cloud.

## Motivation

Data engineering teams on OCI routinely land files in Object Storage as an intermediate step between extract, transform, and load stages. Without a native Kestra integration, teams resort to OCI CLI shell scripts or custom Python tasks, which are hard to template, test, and version alongside the rest of the flow. A native sub-plugin exposes bucket and object operations as typed, expression-aware tasks that integrate naturally with Kestra's internal storage and output system.

## Context

Part of the OCI Plugin Suite EPIC: https://github.com/kestra-io/plugin-oci/issues/2
Reference implementations: `plugin-aws` S3 tasks, `plugin-gcp` GCS tasks.

## API Reference

- **Official docs**: https://docs.oracle.com/en-us/iaas/api/#/en/objectstorage/latest/
- **Authentication**: Config-file (`~/.oci/config`), instance principal, or `SimpleAuthenticationDetailsProvider`
- **Base URL pattern**: `https://objectstorage.{region}.oraclecloud.com/`
- **SDK**: OCI Java SDK v3.87.0 via BOM

## Gradle Dependencies

Add to `build.gradle`:

```groovy
// OCI Java SDK BOM
implementation platform("com.oracle.oci.sdk:oci-java-sdk-bom:3.87.0")
// Object Storage
implementation "com.oracle.oci.sdk:oci-java-sdk-objectstorage"
implementation "com.oracle.oci.sdk:oci-java-sdk-common"
```

## Plugin Structure

- **Repository**: `plugin-oci`
- **Namespace**: `io.kestra.plugin.oci.objectstorage`
- **Sub-plugins**: `objectstorage`

## Suggested Tasks

1. `CreateBucket` — create a new bucket in a namespace
2. `DeleteBucket` — delete an empty bucket
3. `Upload` — upload a file or Kestra internal storage URI to an object
4. `Download` — download an object to Kestra internal storage and expose as output URI
5. `DeleteObject` — delete a specific object by key
6. `ListObjects` — list objects in a bucket with optional prefix filter, emit as output
7. `CopyObject` — copy an object within or across buckets
8. Add `NewObjectTrigger` — poll a bucket prefix for new objects
9. Write unit + integration tests

## YAML Examples

### Example 1 — Upload a processed file to Object Storage

```yaml
id: upload_report_to_oci
namespace: company.data

tasks:
- id: generate_report
type: io.kestra.plugin.core.http.Download
uri: https://internal-api.company.com/report.csv

- id: upload
type: io.kestra.plugin.oci.objectstorage.Upload
region: eu-frankfurt-1
tenancyOcid: "{{ secret('OCI_TENANCY_OCID') }}"
userId: "{{ secret('OCI_USER_OCID') }}"
fingerprint: "{{ secret('OCI_FINGERPRINT') }}"
privateKey: "{{ secret('OCI_PRIVATE_KEY') }}"
namespaceName: "{{ secret('OCI_NAMESPACE') }}"
bucketName: reports
objectName: "reports/{{ execution.startDate | dateFormat('yyyy-MM-dd') }}/report.csv"
from: "{{ outputs.generate_report.uri }}"
```

### Example 2 — List objects and log count

```yaml
id: audit_oci_bucket
namespace: company.data

tasks:
- id: list_objects
type: io.kestra.plugin.oci.objectstorage.ListObjects
region: eu-frankfurt-1
tenancyOcid: "{{ secret('OCI_TENANCY_OCID') }}"
userId: "{{ secret('OCI_USER_OCID') }}"
fingerprint: "{{ secret('OCI_FINGERPRINT') }}"
privateKey: "{{ secret('OCI_PRIVATE_KEY') }}"
namespaceName: "{{ secret('OCI_NAMESPACE') }}"
bucketName: data-lake
prefix: raw/

- id: log_count
type: io.kestra.plugin.core.log.Log
message: "Found {{ outputs.list_objects.count }} objects under raw/"
```

### Example 3 — Trigger a flow when a new object appears in a bucket prefix

```yaml
id: process_new_oci_object
namespace: company.data

triggers:
- id: new_object
type: io.kestra.plugin.oci.objectstorage.NewObjectTrigger
region: eu-frankfurt-1
tenancyOcid: "{{ secret('OCI_TENANCY_OCID') }}"
userId: "{{ secret('OCI_USER_OCID') }}"
fingerprint: "{{ secret('OCI_FINGERPRINT') }}"
privateKey: "{{ secret('OCI_PRIVATE_KEY') }}"
namespaceName: "{{ secret('OCI_NAMESPACE') }}"
bucketName: incoming
prefix: uploads/
interval: PT5M

tasks:
- id: process
type: io.kestra.plugin.core.log.Log
message: "New object: {{ trigger.objectName }} ({{ trigger.size }} bytes)"
```

## Acceptance Criteria

- [ ] `CreateBucket`, `DeleteBucket`, `Upload`, `Download`, `DeleteObject`, `ListObjects`, `CopyObject` tasks implemented
- [ ] `NewObjectTrigger` polling trigger implemented
- [ ] `Upload` accepts both a local path and a Kestra internal storage URI
- [ ] `Download` emits the downloaded object as a Kestra internal storage URI
- [ ] All `Property` fields support Kestra expression language
- [ ] Unit + integration tests pass (`./gradlew test`)
- [ ] `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.CLOUD)`
- [ ] Build passes with `./gradlew build`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with build.gradle and compare the S3 and GCS task implementations in plugin-aws and plugin-gcp. Review the OCI Object Storage API and SDK details in the issue, then implement the listed tasks, trigger, package-info.java, and tests. Done means ./gradlew test and ./gradlew build pass and all acceptance criteria are met.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.