conductor-oss / conductor-oss/conductor
Feature Request: System tasks should support execution log indexing like worker tasks
- Dominant language
- Java
- Stars
- 32.2k
- Forks
- 1k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 41
Description
**Describe the bug**
System tasks (LAMBDA, EVENT, HTTP, etc.) cannot add execution logs during task execution, unlike worker tasks which can add logs via `TaskResult.getLogs()`. This creates a significant gap in Observability and debugging capabilities for system tasks.
**Details**
Conductor version: 3.x
Persistence implementation: Postgres
Queue implementation: Redis
Lock: Redis
Workflow definition: NA
Task definition: NA
Event handler definition: NA
**To Reproduce**
Steps to reproduce the behavior:
1. Create and execute a workflow with a system task (e.g., HTTP)
2. System task executes and completes successfully or fails
3. Try to retrieve execution logs:
- Via UI: Navigate to task execution details → "Logs" tab
- Via API: GET `/api/tasks/{taskId}/log`
4. Observe: No logs are displayed
**Contrast with worker tasks:**
1. Execute a workflow with a worker task
2. Worker adds logs during execution: `taskResult.addLog("Processing step 1...")`
3. Call GET `/api/tasks/{taskId}/log`
4. Result: Logs are returned and visible in UI/API
5. System tasks have no equivalent mechanism
**Expected behavior**
System tasks should also be able to generate execution logs that are:
1. Indexed in the same way as worker task logs (ElasticSearch/OpenSearch/Postgres)
2. Retrievable via the same API: GET `/api/tasks/{taskId}/log`
3. Visible in the Conductor UI under task execution details
**Screenshots**
NA
**Additional context**
**What we've observed in the codebase:**
1. Worker tasks use `TaskResult` which has a `List` logs field
2. When workers call POST `/api/tasks` to update, `WorkflowExecutorOps.updateTask(TaskResult)` processes these logs (line 900-901) and indexes them via `executionDAOFacade.addTaskExecLog(taskResult.getLogs())`
3. System tasks use `TaskModel` which has no logs field
4. When system tasks update via `ExecutionDAOFacade.updateTask(TaskModel)` (line 497), there's no log handling at all
5. This architectural difference means system tasks fundamentally cannot produce execution logs
---
**Note:** If this enhancement is something the community would support, we're happy to discuss design approaches and potentially contribute a PR.
Contributor guide
Research direction
Start with WorkflowExecutorOps.updateTask(TaskResult) around lines 900-901 and ExecutionDAOFacade.updateTask(TaskModel) around line 497, then compare worker log handling with system-task updates. Review TaskResult, TaskModel, and the GET /api/tasks/{taskId}/log path; done means system-task logs can be submitted, indexed, retrieved through the existing API, and shown in the UI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java, postgresql, redis
- Domain
- backend-api-design, databases, distributed-systems, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100