conductor-oss / conductor-oss/conductor

Feature Request: System tasks should support execution log indexing like worker tasks

Open
#597 4 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 5h
Merged PRs (30d)
41

Description

**Describe the bug**
System tasks (LAMBDA, EVENT, HTTP, etc.) cannot add execution logs during task execution, unlike worker tasks which can add logs via `TaskResult.getLogs()`. This creates a significant gap in Observability and debugging capabilities for system tasks.

**Details**
Conductor version: 3.x
Persistence implementation: Postgres
Queue implementation: Redis
Lock: Redis
Workflow definition: NA
Task definition: NA
Event handler definition: NA

**To Reproduce**
Steps to reproduce the behavior:
1. Create and execute a workflow with a system task (e.g., HTTP)
2. System task executes and completes successfully or fails
3. Try to retrieve execution logs:
- Via UI: Navigate to task execution details → "Logs" tab
- Via API: GET `/api/tasks/{taskId}/log`
4. Observe: No logs are displayed

**Contrast with worker tasks:**
1. Execute a workflow with a worker task
2. Worker adds logs during execution: `taskResult.addLog("Processing step 1...")`
3. Call GET `/api/tasks/{taskId}/log`
4. Result: Logs are returned and visible in UI/API
5. System tasks have no equivalent mechanism

**Expected behavior**
System tasks should also be able to generate execution logs that are:
1. Indexed in the same way as worker task logs (ElasticSearch/OpenSearch/Postgres)
2. Retrievable via the same API: GET `/api/tasks/{taskId}/log`
3. Visible in the Conductor UI under task execution details

**Screenshots**
NA

**Additional context**

**What we've observed in the codebase:**
1. Worker tasks use `TaskResult` which has a `List` logs field
2. When workers call POST `/api/tasks` to update, `WorkflowExecutorOps.updateTask(TaskResult)` processes these logs (line 900-901) and indexes them via `executionDAOFacade.addTaskExecLog(taskResult.getLogs())`
3. System tasks use `TaskModel` which has no logs field
4. When system tasks update via `ExecutionDAOFacade.updateTask(TaskModel)` (line 497), there's no log handling at all
5. This architectural difference means system tasks fundamentally cannot produce execution logs

---
**Note:** If this enhancement is something the community would support, we're happy to discuss design approaches and potentially contribute a PR.

Contributor guide

Open the contributing guide

Research direction

Start with WorkflowExecutorOps.updateTask(TaskResult) around lines 900-901 and ExecutionDAOFacade.updateTask(TaskModel) around line 497, then compare worker log handling with system-task updates. Review TaskResult, TaskModel, and the GET /api/tasks/{taskId}/log path; done means system-task logs can be submitted, indexed, retrieved through the existing API, and shown in the UI.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java, postgresql, redis
Domain
backend-api-design, databases, distributed-systems, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.