Elsa Native Background Execution Architecture

## Status

**Draft — Proposed Architecture**

## Purpose

This document describes the **target architecture** for Elsa's background execution subsystem.

The goal is to provide a **workflow-aware, engine-owned background execution system** that replaces reliance on external job runners (such as Hangfire) for internal workflow runtime work.

External schedulers and service buses remain supported but are **not responsible for workflow execution semantics**.

---

# 1. Motivation

Elsa currently relies on pluggable background processing providers (e.g. Hangfire, Quartz, in-memory implementations, service bus consumers).

While flexible, this approach has several drawbacks:

### Lack of workflow awareness

Generic job runners do not understand workflow runtime semantics such as:

* workflow definition lifecycle
* instance lifecycle
* pause / unpause
* drain
* tenant fairness
* workflow correlation
* activity execution context

### Operational inconsistencies

Different providers behave differently regarding:

* retries
* job visibility
* pause semantics
* concurrency
* acquisition

This leads to inconsistent runtime behavior across deployments.

### Inefficient pause semantics

If pause is implemented **after job acquisition**, workers repeatedly acquire jobs that cannot run and must reschedule them, causing churn and unnecessary queue activity.

### Poor runtime observability

Because background work is owned by external systems:

* Elsa cannot provide a complete runtime view of pending work
* operators cannot inspect or manage workflow execution backlog

---

# 2. Design Goals

The Elsa background execution subsystem should:

### 1. Be workflow-aware

Every background job must be associated with workflow runtime context:

* workflow definition
* workflow instance
* activity execution
* tenant
* correlation

### 2. Allow arbitrary job types

Modules must be able to define new job types and handlers.

Example:

```
FooJob
SendWebhookJob
GeneratePdfJob
ResumeWorkflowJob
```

### 3. Support pause without queue churn

Paused workflows must not have their jobs repeatedly dequeued and rescheduled.

Instead:

* jobs remain persisted
* acquisition simply skips them

### 4. Support clustered execution

Multiple workers must safely process jobs concurrently.

### 5. Be provider-independent

Workflow execution semantics must not depend on:

* Hangfire
* MassTransit
* RabbitMQ
* Wolverine
* Quartz

These systems may **feed work into Elsa**, but do not control execution eligibility.

### 6. Provide first-class runtime observability

Operators must be able to inspect:

* pending jobs
* running jobs
* retries
* failures
* backlog

### 7. Maintain modularity

Modules must be able to:

* enqueue jobs
* provide job handlers
* participate in execution

without depending on external job infrastructure.

---

# 3. Non-Goals

This system is **not intended to replace:**

### Service bus infrastructure

External messaging systems remain responsible for:

* cross-service communication
* event-driven architectures
* integration messaging

### Scheduling systems

Cron/timer scheduling remains handled by dedicated schedulers (e.g. Quartz).

### General-purpose application job runners

This system is **not intended as a generic background job framework for arbitrary application tasks**.

All jobs must originate from **workflow execution context**.

---

# 4. Core Concept: Workflow Jobs

A **Workflow Job** represents background work associated with workflow execution.

Unlike generic background jobs, workflow jobs always carry workflow runtime metadata.

Example conceptual structure:

```csharp
WorkflowJob
{
Id
JobType
Payload
WorkflowDefinitionId
WorkflowInstanceId
ActivityId
ActivityNodeId
BookmarkId
CorrelationId
TenantId
}
```

This association enables Elsa to apply runtime policies such as:

* pause
* throttling
* tenant fairness
* workflow lifecycle rules

---

# 5. System Architecture

## High-level architecture

```
External systems
│
▼
Triggers / Activities
│
▼
Workflow Job Scheduler
│
▼
Workflow Job Store (durable)
│
▼
Acquisition Engine
│
▼
Workflow Job Workers
│
▼
Job Handlers
```

---

# 6. Core Components

## 6.1 Workflow Job Scheduler

Responsible for enqueueing jobs.

Example API:

```csharp
public interface IWorkflowJobScheduler
{
Task EnqueueAsync(
string jobType,
object payload,
WorkflowExecutionContextRef context,
WorkflowJobOptions options = null,
CancellationToken cancellationToken = default);
}
```

Responsibilities:

* serialize payload
* store job
* notify workers

---

## 6.2 Workflow Job Store

The durable store of workflow jobs.

Example persistence record:

```csharp
public class WorkflowJobRecord
{
public string Id { get; set; }

public string JobType { get; set; }

public string Payload { get; set; }

public string WorkflowDefinitionId { get; set; }

public string WorkflowInstanceId { get; set; }

public string ActivityId { get; set; }

public string TenantId { get; set; }

public DateTimeOffset CreatedAt { get; set; }

public DateTimeOffset? DueAt { get; set; }

public WorkflowJobStatus Status { get; set; }

public int AttemptCount { get; set; }

public int MaxAttempts { get; set; }

public string LockedBy { get; set; }

public DateTimeOffset? LockExpiresAt { get; set; }

public string LastError { get; set; }
}
```

### Job Status

Minimal initial status model:

```
Pending
Running
Completed
Faulted
```

Paused jobs remain `Pending`.

---

## 6.3 Acquisition Engine

Responsible for selecting runnable jobs.

Workers request jobs using:

```
Acquire runnable jobs
```

Eligibility criteria:

* Status = Pending
* DueAt <= now
* Not locked
* Workflow definition not paused
* Workflow instance not paused
* Tenant enabled

Paused jobs remain persisted but **are not returned by acquisition**.

---

## 6.4 Workflow Job Workers

Workers execute jobs.

Responsibilities:

1. Poll acquisition engine
2. Lease jobs
3. Resolve job handler
4. Execute handler
5. Complete / retry / fail job

Workers can run:

* in-process
* as dedicated worker services
* as distributed nodes

---

## 6.5 Job Handlers

Modules provide job handlers.

Example interface:

```csharp
public interface IWorkflowJobHandler
{
string JobType { get; }

Task ExecuteAsync(
WorkflowJobExecutionContext context,
CancellationToken cancellationToken);
}
```

Handlers should assume:

> If invoked, the job has already been deemed eligible by the engine.

Handlers do not check pause state.

---

# 7. Workflow Context Reference

All jobs carry a reference to workflow execution context.

Example:

```csharp
public record WorkflowExecutionContextRef
(
string WorkflowDefinitionId,
string WorkflowInstanceId,
string ActivityId,
string TenantId,
string CorrelationId
);
```

This enables runtime policies such as:

* pause
* tenant isolation
* concurrency limits
* observability

---

# 8. Pause Semantics

Pause is implemented as an **acquisition rule**, not a worker rule.

When a workflow definition is paused:

* jobs remain `Pending`
* acquisition excludes them
* workers never receive them

When unpaused:

* jobs immediately become eligible

No queue churn occurs.

---

# 9. Distributed Execution

Workers use **lease-based locking**.

Acquisition process:

```
SELECT eligible jobs
SET lock owner
SET lock expiration
```

If a worker crashes:

* lease expires
* another worker may reclaim the job

---

# 10. External System Integration

External systems remain supported but do not control execution semantics.

## Service Bus Integration

Example flow:

```
Message arrives via MassTransit
│
▼
Workflow trigger resolves workflow
│
▼
Enqueue Workflow Job
│
▼
Elsa Worker executes job
```

## Scheduler Integration

Example flow:

```
Cron fires (Quartz)
│
▼
Workflow start trigger
│
▼
Enqueue Workflow Job
```

---

# 11. Observability

The job store enables first-class runtime visibility.

Operators can inspect:

* pending jobs
* running jobs
* failures
* retries
* backlog

Future UI features may include:

* retry job
* cancel job
* unlock job
* replay job

---

# 12. Job Categories (Future)

Optional classification:

```
Continuation
ActivityWork
Integration
Maintenance
Custom
```

This allows:

* prioritization
* throttling
* monitoring

---

# 13. Execution Pipeline

Worker pipeline:

```
Acquire jobs
│
Check eligibility
│
Lease jobs
│
Resolve handler
│
Execute
│
Complete / retry / fail
```

---

# 14. Phased Implementation Plan

## Phase 1 — Minimal Execution Engine

Deliver core infrastructure.

Features:

* workflow job store
* enqueue API
* polling worker
* lease-based locking
* handler resolution
* job completion
* basic retry

Goal:

Replace in-memory background execution.

---

## Phase 2 — Pause-Aware Execution

Add pause semantics.

Features:

* workflow pause support
* acquisition filtering
* worker safety gate
* runtime APIs

Goal:

Allow safe workflow pause without job churn.

---

## Phase 3 — Operational Controls

Introduce runtime management features.

Features:

* drain mode
* retry policies
* dead-letter jobs
* job inspection APIs
* runtime metrics

Goal:

Operational stability.

---

## Phase 4 — Scalability

Improve distributed behavior.

Features:

* better acquisition batching
* partitioned work
* tenant fairness
* concurrency limits

Goal:

Large-scale deployments.

---

## Phase 5 — Advanced Runtime

Future capabilities.

Features:

* priority scheduling
* work partitioning
* wake-up signals
* adaptive throttling
* UI tooling

---

# 15. Summary

The Elsa native background execution system provides:

* engine-owned workflow job execution
* pause-aware scheduling
* distributed worker coordination
* extensible module job handlers
* runtime observability

External transports and schedulers remain supported but are no longer responsible for workflow runtime semantics.