kestra-io / kestra-io/plugin-documentdb

Migrate from REST HTTP to official MongoDB Java Driver (org.mongodb:mongodb-driver-sync)

Open
#34 0 comments 0 reactions 0 assignees View on GitHub
area/plugin kind/breaking-change kind/cooldown
Dominant language
Java
Stars
0
Forks
1
Avg merge
1d 12h
Merged PRs (30d)
4

Description

## Summary

The plugin currently performs CRUD operations against Amazon DocumentDB via its REST API using Kestra's internal HTTP client. Since Amazon DocumentDB is MongoDB-compatible, the correct and AWS-recommended integration path is the official MongoDB Java Driver — `org.mongodb:mongodb-driver-sync` (v5.5.1, released 2025-06-06, 118 versions). Migrating to the driver provides a far richer API (aggregation pipelines, transactions, change streams, typed codec support) and eliminates HTTP-level JSON serialisation entirely.

## Motivation

- Amazon DocumentDB exposes a MongoDB-compatible wire protocol; the MongoDB Java Driver is the client AWS itself recommends in its DocumentDB documentation.
- Raw REST API access to DocumentDB is an unusual integration path that limits available operations to what the REST layer exposes. The native driver unlocks the full MongoDB API surface.
- The MongoDB Java Driver handles connection pooling, TLS/SSL, retries, read preferences, and write concerns out of the box — none of which the current HTTP client implementation handles.
- The driver is extremely actively maintained (118 versions, v5.5.1 in June 2025) and versioned in lockstep with the MongoDB server protocol.

## Context

Current implementation uses `io.kestra.core.http.client.HttpClient` against a DocumentDB REST API endpoint. The migration should replace REST calls with `MongoClient` and `MongoCollection` operations, connecting via the standard MongoDB connection string (`mongodb://...` or `mongodb+srv://...` for DocumentDB).

AWS DocumentDB guide: https://docs.aws.amazon.com/documentdb/latest/developerguide/connect-from-outside-a-vpc.html
MongoDB Java Driver: https://www.mongodb.com/docs/drivers/java/sync/current/

## API Reference

- **Official docs**: https://docs.aws.amazon.com/documentdb/latest/developerguide/developer-guide.html
- **Authentication**: Username/password via connection string, TLS via `tls=true&tlsCAFile=...` parameter
- **Connection string**: `mongodb://:@:27017/?tls=true&...`
- **Driver**: `org.mongodb:mongodb-driver-sync` v5.5.1

## Gradle Dependencies

```groovy
// MongoDB Java Driver (sync) — compatible with Amazon DocumentDB
implementation "org.mongodb:mongodb-driver-sync:5.5.1"
```

## Plugin Structure

- **Repository**: `kestra-io/plugin-documentdb`
- **Namespace**: `io.kestra.plugin.documentdb`
- **Categories**: `DATA`, `CLOUD`

## Suggested Tasks

1. Add `org.mongodb:mongodb-driver-sync` to `build.gradle`; remove any HTTP client dependencies used solely for DocumentDB REST calls
2. Implement a shared `DocumentDbClientFactory` or abstract base task that builds a `MongoClient` from `Property connectionString` (or individual host/port/credentials), handling TLS options for DocumentDB
3. Annotate credentials in the connection string (or `Property password`) with `@PluginProperty(secret = true)`
4. Implement or migrate tasks: `InsertOne`, `InsertMany`, `FindOne`, `Find`, `UpdateOne`, `UpdateMany`, `DeleteOne`, `DeleteMany`, `Aggregate`
5. Support `fetchType` (`FETCH_ONE`, `FETCH`, `STORE`) for read tasks returning data
6. Use `Property` for all properties; model query/pipeline as `Property>`
7. Update unit tests to use a Testcontainers MongoDB instance (compatible with DocumentDB wire protocol) or Wiremock if Testcontainers is unavailable
8. Verify all `@Schema` annotations, Lombok annotations, and logging via `runContext.logger()` are present
9. Update YAML examples and documentation

## YAML Examples

### Example 1 — Insert a document into a DocumentDB collection

```yaml
id: insert_documentdb
namespace: company.team

tasks:
- id: insert
type: io.kestra.plugin.documentdb.Insert
connectionString: "{{ secret('DOCUMENTDB_URI') }}"
database: mydb
collection: events
document:
eventId: "{{ execution.id }}"
status: completed
timestamp: "{{ execution.startDate }}"
```

### Example 2 — Query documents and store results

```yaml
id: query_documentdb
namespace: company.team

tasks:
- id: find_records
type: io.kestra.plugin.documentdb.Find
connectionString: "{{ secret('DOCUMENTDB_URI') }}"
database: mydb
collection: orders
filter:
status: pending
fetchType: STORE

- id: log_count
type: io.kestra.plugin.core.log.Log
message: "Found {{ outputs.find_records.size }} pending orders"
```

### Example 3 — Trigger on new documents inserted into a collection

```yaml
id: react_to_new_document
namespace: company.team

triggers:
- id: on_new_order
type: io.kestra.plugin.documentdb.InsertTrigger
connectionString: "{{ secret('DOCUMENTDB_URI') }}"
database: mydb
collection: orders
filter:
status: new
interval: PT1M

tasks:
- id: process_order
type: io.kestra.plugin.core.log.Log
message: "New order received: {{ trigger.documentId }}"
```

## Acceptance Criteria

### Functional
- [ ] CRUD operations work against a DocumentDB-compatible MongoDB instance
- [ ] TLS/SSL configuration supported via connection string parameters
- [ ] `fetchType` (`FETCH_ONE`, `FETCH`, `STORE`) supported on read tasks
- [ ] Unit tests pass (`./gradlew test`) using Testcontainers or Wiremock
- [ ] Build passes (`./gradlew build`)

### Kestra Plugin Coding Standards
- [ ] All new/refactored properties use `Property`
- [ ] Credentials annotated with `@PluginProperty(secret = true)`
- [ ] Every property and output has a `@Schema` annotation
- [ ] Task classes carry all five mandatory Lombok annotations
- [ ] Logging via `runContext.logger()` only
- [ ] No raw HTTP client dependencies remain for DocumentDB operations

### Documentation
- [ ] `@Plugin(examples = ...)` entries set `full = true` with complete runnable flows
- [ ] Sensitive values in examples use `{{ secret('...') }}`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with build.gradle and the existing DocumentDB task implementations to map the REST client usage and shared properties. Review the CRUD and trigger entry points, then run ./gradlew test to understand current coverage. Done means MongoDB-driver operations cover the listed tasks, tests and build pass, and examples and documentation meet the stated acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.