kestra-io / kestra-io/plugin-oci
[Plugin] OCI — Streaming (Kafka-Compatible)
- Dominant language
- Java
- Stars
- 0
- Forks
- 0
- Avg merge
- 18h 3m
- Merged PRs (30d)
- 1
Description
## Summary
The OCI Streaming sub-plugin for `plugin-oci` enables Kestra flows to produce and consume messages on OCI Streaming — Oracle's Kafka-compatible managed streaming service. This allows flows to publish events to streams as part of data pipelines, and to trigger flows reactively when new messages arrive, bridging event-driven and batch orchestration on OCI.
## Motivation
Teams building event-driven architectures on OCI Streaming face a challenge: they can produce messages via Kafka clients, but orchestrating the produce step as a typed Kestra task — with expression-aware payload templating, retry, and structured output — requires a custom HTTP or Kafka task. A native OCI Streaming sub-plugin handles the OCI-specific authentication and stream cursor management transparently, making produce/consume operations first-class Kestra tasks.
## Context
Part of the OCI Plugin Suite EPIC: https://github.com/kestra-io/plugin-oci/issues/2
OCI Streaming is Kafka-compatible; however, the OCI Java SDK provides a native client (`StreamClient`, `StreamAdminClient`) that handles OCI request signing, which is preferable to a raw Kafka client for authentication reasons.
Reference: `plugin-kafka` producer/consumer pattern.
## API Reference
- **Official docs**: https://docs.oracle.com/en-us/iaas/api/#/en/streaming/latest/
- **Authentication**: Config-file (`~/.oci/config`), instance principal, or `SimpleAuthenticationDetailsProvider`
- **Base URL pattern**: `https://streaming.{region}.oci.oraclecloud.com/20180418/`
- **SDK**: OCI Java SDK v3.87.0 via BOM
## Gradle Dependencies
Add to `build.gradle`:
```groovy
// OCI Java SDK BOM
implementation platform("com.oracle.oci.sdk:oci-java-sdk-bom:3.87.0")
// Streaming
implementation "com.oracle.oci.sdk:oci-java-sdk-streaming"
implementation "com.oracle.oci.sdk:oci-java-sdk-common"
```
## Plugin Structure
- **Repository**: `plugin-oci`
- **Namespace**: `io.kestra.plugin.oci.streaming`
- **Sub-plugins**: `streaming` (produce, consume, admin)
## Suggested Tasks
1. `Produce` — publish one or more messages to a stream (with key, value, and partition key)
2. `Consume` — consume messages from a stream using a group cursor, emit as output
3. `CreateStream` — create a new stream in a compartment
4. `DeleteStream` — delete a stream
5. `ListStreams` — list streams in a compartment
6. `GetStream` — fetch stream details and emit the messages endpoint as output
7. Add `NewMessageTrigger` — poll a stream for new messages and trigger a flow per batch
8. Write unit + integration tests
## YAML Examples
### Example 1 — Publish a pipeline completion event to a stream
```yaml
id: publish_pipeline_event
namespace: company.platform
tasks:
- id: run_pipeline
type: io.kestra.plugin.core.log.Log
message: Running pipeline...
- id: publish_event
type: io.kestra.plugin.oci.streaming.Produce
region: eu-frankfurt-1
tenancyOcid: "{{ secret('OCI_TENANCY_OCID') }}"
userId: "{{ secret('OCI_USER_OCID') }}"
fingerprint: "{{ secret('OCI_FINGERPRINT') }}"
privateKey: "{{ secret('OCI_PRIVATE_KEY') }}"
streamId: "{{ secret('OCI_STREAM_OCID') }}"
messages:
- key: pipeline-complete
value: "{{ execution.id }}"
```
### Example 2 — Consume messages and log count
```yaml
id: consume_oci_stream
namespace: company.platform
tasks:
- id: consume
type: io.kestra.plugin.oci.streaming.Consume
region: eu-frankfurt-1
tenancyOcid: "{{ secret('OCI_TENANCY_OCID') }}"
userId: "{{ secret('OCI_USER_OCID') }}"
fingerprint: "{{ secret('OCI_FINGERPRINT') }}"
privateKey: "{{ secret('OCI_PRIVATE_KEY') }}"
streamId: "{{ secret('OCI_STREAM_OCID') }}"
groupName: kestra-consumer-group
maxMessages: 100
- id: log_count
type: io.kestra.plugin.core.log.Log
message: "Consumed {{ outputs.consume.count }} messages"
```
### Example 3 — Trigger a flow when new messages arrive on a stream
```yaml
id: on_new_stream_messages
namespace: company.platform
triggers:
- id: stream_watcher
type: io.kestra.plugin.oci.streaming.NewMessageTrigger
region: eu-frankfurt-1
tenancyOcid: "{{ secret('OCI_TENANCY_OCID') }}"
userId: "{{ secret('OCI_USER_OCID') }}"
fingerprint: "{{ secret('OCI_FINGERPRINT') }}"
privateKey: "{{ secret('OCI_PRIVATE_KEY') }}"
streamId: "{{ secret('OCI_STREAM_OCID') }}"
groupName: kestra-trigger-group
maxMessages: 50
interval: PT1M
tasks:
- id: handle
type: io.kestra.plugin.core.log.Log
message: "Received {{ trigger.count }} messages from OCI Streaming"
```
## Acceptance Criteria
- [ ] `Produce`, `Consume`, `CreateStream`, `DeleteStream`, `ListStreams`, `GetStream` tasks implemented
- [ ] `NewMessageTrigger` polling trigger implemented
- [ ] `Produce` supports batch publishing (list of key/value messages)
- [ ] `Consume` uses group cursor and emits messages as structured output
- [ ] All `Property` fields support Kestra expression language
- [ ] Unit + integration tests pass (`./gradlew test`)
- [ ] `package-info.java` with `@PluginSubGroup(category = PluginSubGroup.PluginCategory.CLOUD)`
- [ ] Build passes with `./gradlew build`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the plugin-kafka producer/consumer pattern and the OCI SDK dependencies proposed for build.gradle. Implement the listed streaming tasks and NewMessageTrigger under io.kestra.plugin.oci.streaming, then run ./gradlew test and ./gradlew build; done means the acceptance checklist is satisfied, including expression-aware fields and unit plus integration tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- cloud, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100