kestra-io / kestra-io/plugin-oci
[Plugin] OCI — Object Storage (Buckets & Objects)
- Dominant language
- Java
- Stars
- 0
- Forks
- 0
- Avg merge
- 18h 3m
- Merged PRs (30d)
- 1
Description
## Summary
The OCI Object Storage sub-plugin for `plugin-oci` lets Kestra flows create and manage buckets, upload and download objects, list bucket contents, and delete objects — all natively within a flow. It turns OCI Object Storage into a first-class data handoff layer between tasks, complementing the existing S3 and GCS plugins for teams standardized on Oracle Cloud.
## Motivation
Data engineering teams on OCI routinely land files in Object Storage as an intermediate step between extract, transform, and load stages. Without a native Kestra integration, teams resort to OCI CLI shell scripts or custom Python tasks, which are hard to template, test, and version alongside the rest of the flow. A native sub-plugin exposes bucket and object operations as typed, expression-aware tasks that integrate naturally with Kestra's internal storage and output system.
## Context
Part of the OCI Plugin Suite EPIC: https://github.com/kestra-io/plugin-oci/issues/2
Reference implementations: `plugin-aws` S3 tasks, `plugin-gcp` GCS tasks.
## API Reference
- **Official docs**: https://docs.oracle.com/en-us/iaas/api/#/en/objectstorage/latest/
- **Authentication**: Config-file (`~/.oci/config`), instance principal, or `SimpleAuthenticationDetailsProvider`
- **Base URL pattern**: `https://objectstorage.{region}.oraclecloud.com/`
- **SDK**: OCI Java SDK v3.87.0 via BOM
## Gradle Dependencies
Add to `build.gradle`:
```groovy
// OCI Java SDK BOM
implementation platform("com.oracle.oci.sdk:oci-java-sdk-bom:3.87.0")
// Object Storage
implementation "com.oracle.oci.sdk:oci-java-sdk-objectstorage"
implementation "com.oracle.oci.sdk:oci-java-sdk-common"
```
## Plugin Structure
- **Repository**: `plugin-oci`
- **Namespace**: `io.kestra.plugin.oci.objectstorage`
- **Sub-plugins**: `objectstorage`
## Suggested Tasks
1. `CreateBucket` — create a new bucket in a namespace
2. `DeleteBucket` — delete an empty bucket
3. `Upload` — upload a file or Kestra internal storage URI to an object
4. `Download` — download an object to Kestra internal storage and expose as output URI
5. `DeleteObject` — delete a specific object by key
6. `ListObjects` — list objects in a bucket with optional prefix filter, emit as output
7. `CopyObject` — copy an object within or across buckets
8. Add `NewObjectTrigger` — poll a bucket prefix for new objects
9. Write unit + integration tests
## YAML Examples
### Example 1 — Upload a processed file to Object Storage
```yaml
id: upload_report_to_oci
namespace: company.data
tasks:
- id: generate_report
type: io.kestra.plugin.core.http.Download
uri: https://internal-api.company.com/report.csv
- id: upload
type: io.kestra.plugin.oci.objectstorage.Upload
region: eu-frankfurt-1
tenancyOcid: "{{ secret('OCI_TENANCY_OCID') }}"
userId: "{{ secret('OCI_USER_OCID') }}"
fingerprint: "{{ secret('OCI_FINGERPRINT') }}"
privateKey: "{{ secret('OCI_PRIVATE_KEY') }}"
namespaceName: "{{ secret('OCI_NAMESPACE') }}"
bucketName: reports
objectName: "reports/{{ execution.startDate | dateFormat('yyyy-MM-dd') }}/report.csv"
from: "{{ outputs.generate_report.uri }}"
```
### Example 2 — List objects and log count
```yaml
id: audit_oci_bucket
namespace: company.data
tasks:
- id: list_objects
type: io.kestra.plugin.oci.objectstorage.ListObjects
region: eu-frankfurt-1
tenancyOcid: "{{ secret('OCI_TENANCY_OCID') }}"
userId: "{{ secret('OCI_USER_OCID') }}"
fingerprint: "{{ secret('OCI_FINGERPRINT') }}"
privateKey: "{{ secret('OCI_PRIVATE_KEY') }}"
namespaceName: "{{ secret('OCI_NAMESPACE') }}"
bucketName: data-lake
prefix: raw/
- id: log_count
type: io.kestra.plugin.core.log.Log
message: "Found {{ outputs.list_objects.count }} objects under raw/"
```
### Example 3 — Trigger a flow when a new object appears in a bucket prefix
```yaml
id: process_new_oci_object
namespace: company.data
triggers:
- id: new_object
type: io.kestra.plugin.oci.objectstorage.NewObjectTrigger
region: eu-frankfurt-1
tenancyOcid: "{{ secret('OCI_TENANCY_OCID') }}"
userId: "{{ secret('OCI_USER_OCID') }}"
fingerprint: "{{ secret('OCI_FINGERPRINT') }}"
privateKey: "{{ secret('OCI_PRIVATE_KEY') }}"
namespaceName: "{{ secret('OCI_NAMESPACE') }}"
bucketName: incoming
prefix: uploads/
interval: PT5M
tasks:
- id: process
type: io.kestra.plugin.core.log.Log
message: "New object: {{ trigger.objectName }} ({{ trigger.size }} bytes)"
```
## Acceptance Criteria
- [ ] `CreateBucket`, `DeleteBucket`, `Upload`, `Download`, `DeleteObject`, `ListObjects`, `CopyObject` tasks implemented
- [ ] `NewObjectTrigger` polling trigger implemented
- [ ] `Upload` accepts both a local path and a Kestra internal storage URI
- [ ] `Download` emits the downloaded object as a Kestra internal storage URI
- [ ] All `Property` fields support Kestra expression language
- [ ] Unit + integration tests pass (`./gradlew test`)
- [ ] `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.CLOUD)`
- [ ] Build passes with `./gradlew build`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with build.gradle and compare the S3 and GCS task implementations in plugin-aws and plugin-gcp. Review the OCI Object Storage API and SDK details in the issue, then implement the listed tasks, trigger, package-info.java, and tests. Done means ./gradlew test and ./gradlew build pass and all acceptance criteria are met.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100