conductor-oss / conductor-oss/conductor

Epic: Implement idempotency support for SUB_WORKFLOW (Q4 Roadmap 2.1)

Open
#622 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 5h
Merged PRs (30d)
41

Description

# SUB_WORKFLOW: Implement Idempotency Support

## Context

SUB_WORKFLOW lacks an idempotency mechanism, leading to duplicate subworkflow executions in production scenarios. This creates data integrity issues, duplicate processing, and potential double-charging scenarios in retry or failure cases. Idempotency is a fundamental requirement for reliable workflow patterns in distributed systems where retries are common.

## Current State vs. Desired State

**Current:**
- No idempotency mechanism for SUB_WORKFLOW tasks
- Retries can cause duplicate subworkflow executions
- Data integrity issues from duplicate processing
- Potential double-charging in payment/billing scenarios
- No way to guarantee "execute exactly once" semantics

**Desired:**
- Configurable idempotency via `idempotencyKey` parameter
- "Execute exactly once" semantics for idempotent subworkflows
- Duplicate execution prevention based on idempotency key
- Support for parameterized keys using workflow expressions
- Minimal performance overhead for non-idempotent workflows

## Technical Scope

### What Needs to Be Implemented

- [ ] Add `idempotencyKey` parameter to SUB_WORKFLOW task definition
- [ ] Implement idempotency tracking mechanism (using workflow ID + key)
- [ ] Add deduplication logic before subworkflow execution
- [ ] Support parameterized idempotency keys using workflow expressions
- [ ] Handle idempotency key collisions appropriately
- [ ] Add configuration for idempotency behavior (TTL, storage backend)

### Implementation Approach

Port idempotency support from Enterprise. Add `idempotencyKey` parameter to SUB_WORKFLOW task definition. Implement tracking mechanism to detect duplicate execution requests. Return existing subworkflow execution when idempotency key matches.

### Files/Components Affected

- `SubWorkflow.java` (add idempotency logic ~100 lines)
- Idempotency tracking DAO (new/updated)
- Task definition schema (add `idempotencyKey` parameter)
- Workflow execution engine (deduplication logic)
- Configuration (idempotency settings)

### Dependencies

- No blocking dependencies on other issues
- Critical foundation for reliable workflow patterns in Phase 4 (AI workflows)

## Success Criteria

- [ ] "Execute exactly once" semantics for idempotent subworkflows
- [ ] Duplicate execution requests with matching keys are prevented
- [ ] Existing subworkflow execution returned when idempotency key matches
- [ ] Parameterized idempotency keys work with workflow expressions
- [ ] Performance overhead is minimal (<5% for non-idempotent workflows)
- [ ] Feature is opt-in via `idempotencyKey` parameter
- [ ] Backward compatibility maintained (workflows without key behave as before)
- [ ] Configurable TTL prevents unbounded storage growth

## Testing Requirements

- [ ] Unit tests for idempotency key parsing
- [ ] Unit tests for idempotency tracking mechanism
- [ ] Integration tests with duplicate execution attempts
- [ ] Integration tests with parameterized keys using expressions
- [ ] Concurrent execution tests for race conditions
- [ ] Database compatibility tests (all supported databases)
- [ ] Performance benchmarks (with and without idempotency)
- [ ] Backward compatibility verification
- [ ] TTL expiration tests for key cleanup

## Documentation Needs

- [ ] Documentation for `idempotencyKey` parameter
- [ ] Idempotency patterns guide (when and how to use)
- [ ] Configuration reference for TTL and storage settings
- [ ] Examples for payment processing workflows
- [ ] Examples for API integration workflows
- [ ] Release notes highlighting idempotency support
- [ ] Best practices for choosing idempotency keys
- [ ] Troubleshooting guide for idempotency issues

## Scope Notes

**Opt-in Feature:** Workflows without `idempotencyKey` behave exactly as before. This ensures zero breaking changes and backward compatibility.

**Key Scoping:** Consider using `workflowId + idempotencyKey` as the deduplication identifier to scope idempotency per parent workflow, preventing cross-workflow collisions.

**TTL Configuration:** Idempotency keys should have configurable TTL to prevent unbounded storage growth over time.

**Use Cases:** Critical for payment processing, API integrations, and any scenario where duplicate execution creates data integrity or financial issues.

Contributor guide

Open the contributing guide

Research direction

Start by locating SubWorkflow.java, the task definition schema, the idempotency tracking DAO, and the workflow execution engine. Read the existing Enterprise idempotency implementation first, then review integration and concurrency test locations; done means duplicate keyed executions are prevented, unkeyed behavior remains unchanged, and TTL and database compatibility are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.