kestra-io / kestra-io/plugin-documentdb
Migrate from REST HTTP to official MongoDB Java Driver (org.mongodb:mongodb-driver-sync)
- Dominant language
- Java
- Stars
- 0
- Forks
- 1
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 4
Description
## Summary
The plugin currently performs CRUD operations against Amazon DocumentDB via its REST API using Kestra's internal HTTP client. Since Amazon DocumentDB is MongoDB-compatible, the correct and AWS-recommended integration path is the official MongoDB Java Driver — `org.mongodb:mongodb-driver-sync` (v5.5.1, released 2025-06-06, 118 versions). Migrating to the driver provides a far richer API (aggregation pipelines, transactions, change streams, typed codec support) and eliminates HTTP-level JSON serialisation entirely.
## Motivation
- Amazon DocumentDB exposes a MongoDB-compatible wire protocol; the MongoDB Java Driver is the client AWS itself recommends in its DocumentDB documentation.
- Raw REST API access to DocumentDB is an unusual integration path that limits available operations to what the REST layer exposes. The native driver unlocks the full MongoDB API surface.
- The MongoDB Java Driver handles connection pooling, TLS/SSL, retries, read preferences, and write concerns out of the box — none of which the current HTTP client implementation handles.
- The driver is extremely actively maintained (118 versions, v5.5.1 in June 2025) and versioned in lockstep with the MongoDB server protocol.
## Context
Current implementation uses `io.kestra.core.http.client.HttpClient` against a DocumentDB REST API endpoint. The migration should replace REST calls with `MongoClient` and `MongoCollection` operations, connecting via the standard MongoDB connection string (`mongodb://...` or `mongodb+srv://...` for DocumentDB).
AWS DocumentDB guide: https://docs.aws.amazon.com/documentdb/latest/developerguide/connect-from-outside-a-vpc.html
MongoDB Java Driver: https://www.mongodb.com/docs/drivers/java/sync/current/
## API Reference
- **Official docs**: https://docs.aws.amazon.com/documentdb/latest/developerguide/developer-guide.html
- **Authentication**: Username/password via connection string, TLS via `tls=true&tlsCAFile=...` parameter
- **Connection string**: `mongodb://:@:27017/?tls=true&...`
- **Driver**: `org.mongodb:mongodb-driver-sync` v5.5.1
## Gradle Dependencies
```groovy
// MongoDB Java Driver (sync) — compatible with Amazon DocumentDB
implementation "org.mongodb:mongodb-driver-sync:5.5.1"
```
## Plugin Structure
- **Repository**: `kestra-io/plugin-documentdb`
- **Namespace**: `io.kestra.plugin.documentdb`
- **Categories**: `DATA`, `CLOUD`
## Suggested Tasks
1. Add `org.mongodb:mongodb-driver-sync` to `build.gradle`; remove any HTTP client dependencies used solely for DocumentDB REST calls
2. Implement a shared `DocumentDbClientFactory` or abstract base task that builds a `MongoClient` from `Property connectionString` (or individual host/port/credentials), handling TLS options for DocumentDB
3. Annotate credentials in the connection string (or `Property password`) with `@PluginProperty(secret = true)`
4. Implement or migrate tasks: `InsertOne`, `InsertMany`, `FindOne`, `Find`, `UpdateOne`, `UpdateMany`, `DeleteOne`, `DeleteMany`, `Aggregate`
5. Support `fetchType` (`FETCH_ONE`, `FETCH`, `STORE`) for read tasks returning data
6. Use `Property` for all properties; model query/pipeline as `Property>`
7. Update unit tests to use a Testcontainers MongoDB instance (compatible with DocumentDB wire protocol) or Wiremock if Testcontainers is unavailable
8. Verify all `@Schema` annotations, Lombok annotations, and logging via `runContext.logger()` are present
9. Update YAML examples and documentation
## YAML Examples
### Example 1 — Insert a document into a DocumentDB collection
```yaml
id: insert_documentdb
namespace: company.team
tasks:
- id: insert
type: io.kestra.plugin.documentdb.Insert
connectionString: "{{ secret('DOCUMENTDB_URI') }}"
database: mydb
collection: events
document:
eventId: "{{ execution.id }}"
status: completed
timestamp: "{{ execution.startDate }}"
```
### Example 2 — Query documents and store results
```yaml
id: query_documentdb
namespace: company.team
tasks:
- id: find_records
type: io.kestra.plugin.documentdb.Find
connectionString: "{{ secret('DOCUMENTDB_URI') }}"
database: mydb
collection: orders
filter:
status: pending
fetchType: STORE
- id: log_count
type: io.kestra.plugin.core.log.Log
message: "Found {{ outputs.find_records.size }} pending orders"
```
### Example 3 — Trigger on new documents inserted into a collection
```yaml
id: react_to_new_document
namespace: company.team
triggers:
- id: on_new_order
type: io.kestra.plugin.documentdb.InsertTrigger
connectionString: "{{ secret('DOCUMENTDB_URI') }}"
database: mydb
collection: orders
filter:
status: new
interval: PT1M
tasks:
- id: process_order
type: io.kestra.plugin.core.log.Log
message: "New order received: {{ trigger.documentId }}"
```
## Acceptance Criteria
### Functional
- [ ] CRUD operations work against a DocumentDB-compatible MongoDB instance
- [ ] TLS/SSL configuration supported via connection string parameters
- [ ] `fetchType` (`FETCH_ONE`, `FETCH`, `STORE`) supported on read tasks
- [ ] Unit tests pass (`./gradlew test`) using Testcontainers or Wiremock
- [ ] Build passes (`./gradlew build`)
### Kestra Plugin Coding Standards
- [ ] All new/refactored properties use `Property`
- [ ] Credentials annotated with `@PluginProperty(secret = true)`
- [ ] Every property and output has a `@Schema` annotation
- [ ] Task classes carry all five mandatory Lombok annotations
- [ ] Logging via `runContext.logger()` only
- [ ] No raw HTTP client dependencies remain for DocumentDB operations
### Documentation
- [ ] `@Plugin(examples = ...)` entries set `full = true` with complete runnable flows
- [ ] Sensitive values in examples use `{{ secret('...') }}`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with build.gradle and the existing DocumentDB task implementations to map the REST client usage and shared properties. Review the CRUD and trigger entry points, then run ./gradlew test to understand current coverage. Done means MongoDB-driver operations cover the listed tasks, tests and build pass, and examples and documentation meet the stated acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100