conductor-oss / conductor-oss/conductor

[FEATURE] Support for Administrative Re-indexing / Data Synchronization from Persistence (Postgres) to Indexing (ES/OpenSearch)

Open
#875 0 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 3h
Merged PRs (30d)
37

Description

### The feature, motivation and pitch

Problem Description
Currently, Conductor relies on event-driven indexing. However, in several production scenarios, the search index can become out-of-sync with the primary database (Postgres):

Data Loss/Corruption: Partial or total data loss in Elasticsearch/OpenSearch.

Schema/Configuration Changes: When changing index mappings or upgrading search engine versions.

Missing Events: Occasional failures in the indexing pipeline that leave gaps in the search results.

There is currently no native mechanism to "replay" or bulk-sync historical data from the primary ExecutionDAO (Postgres) to the IndexDAO.

Proposed Feature
I propose adding an administrative tool or API that can:

Scan & Repair: Iterate through workflow and task records in Postgres.

Bulk Re-index: Push missing or outdated records to the indexing layer (ES/OpenSearch).

Filtering: Support re-indexing by specific time ranges or workflow types to avoid full-cluster load.

Implementation Plan (Draft)
Extend the ExecutionDAO to support a paginated stream of historical records.

Implement a service/task (perhaps a specialized System Task or an Admin API endpoint) that calls the IndexDAO for each record.

Ensure the process is throttled to prevent performance degradation of the live system.

Contribution
I am very interested in this feature and would like to participate in the development. I can help with the design, implementation, and testing of the synchronization logic. Please let me know if this aligns with the project roadmap, and I’ll be happy to start a PR!

### Alternatives

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the ExecutionDAO and IndexDAO mentioned in the proposal, then determine how historical workflow and task records are currently accessed and indexed. The feature is complete when an agreed administrative mechanism supports paginated synchronization with time-range or workflow-type filtering and throttling; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java, postgresql
Domain
backend-api-design, databases, distributed-systems, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.