conductor-oss / conductor-oss/conductor

MySQL backend without Elasticsearch: workflow search silently returns nothing

Open
#1,602 1 comment 0 reactions 0 assignees View on GitHub
documentation enhancement
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 5h
Merged PRs (30d)
41

Description

## What happens

On a MySQL deployment with no Elasticsearch/OpenSearch (indexing disabled), workflow executions never show up in search. `GET /api/workflow/search` returns `totalHits=0` for every workflow (including ones you just started), while `GET /api/workflow/{id}` returns them just fine. It is a silent empty (HTTP 200, no error), so from the outside it looks like your executions vanished.

Verified on (prospective) 3.32.2.

## Why

Search is served by an `IndexDAO` bean, and on MySQL your only real options are external ES/OpenSearch or nothing:

- With `conductor.indexing.enabled=false`, the wired bean is `NoopIndexDAO`, whose `search*` methods always return `SearchResult(0, [])`.
- `SqliteIndexDAO` (the embedded fallback that does work) is gated on `conductor.db.type=sqlite` (its `SqliteConfiguration` is `@ConditionalOnProperty(conductor.db.type=sqlite)`), so a MySQL deployment cannot use it.
- There is no MySQL-native `IndexDAO` (unlike Postgres, which gets `PostgresIndexDAO` for DB-native search without ES).

Net: a MySQL deployment without ES has no working search backend, and the failure is silent.

## Repro

Stack: MySQL (persistence) + Redis (queues), no Elasticsearch. Set `conductor.db.type=mysql` and `conductor.indexing.enabled=false`.

```bash
API=http://localhost:8080/api

# register a workflow that auto-completes (single SET_VARIABLE task, no worker needed)
curl -s -X PUT "$API/metadata/workflow" -H 'Content-Type: application/json' -d '[{
"name":"mysql_noes","version":1,"ownerEmail":"t@t.co","timeoutSeconds":60,
"tasks":[{"name":"setv","taskReferenceName":"setv","type":"SET_VARIABLE","inputParameters":{"x":1}}]
}]'

# start it
wid=$(curl -s -X POST "$API/workflow/mysql_noes?version=1" -H 'Content-Type: application/json' -d '{}')

# fetch by id -- works (persistence is fine)
curl -s -o /dev/null -w '%{http_code}\n' "$API/workflow/$wid" # -> 200

# search for it -- empty
curl -s "$API/workflow/search?freeText=mysql_noes" # -> {"totalHits":0,"results":[]}
```

### Config gotcha (worth calling out)

You have to set `conductor.indexing.enabled=false` in the properties file, not via `-e` / `-D`. `Conductor.loadExternalConfig()` copies every property from `CONDUCTOR_CONFIG_FILE` into system properties, so the shipped `config-mysql.properties` line `conductor.indexing.enabled=true` overrides anything you pass on the command line.

## Suggestions (any subset)

- Docs: state clearly that MySQL requires ES/OpenSearch for search (there is no MySQL-native search, unlike Postgres).
- UX: log a startup WARN (and/or surface a hint in search responses) when the active `IndexDAO` is `NoopIndexDAO`, so an empty search is not silent.
- Feature (larger): a MySQL-native `IndexDAO` mirroring `PostgresIndexDAO`.

Contributor guide

Open the contributing guide

Research direction

Start with the IndexDAO implementations, especially NoopIndexDAO and PostgresIndexDAO, then inspect SqliteConfiguration and Conductor.loadExternalConfig() alongside config-mysql.properties. Reproduce the workflow search request with MySQL and indexing disabled. Done means the agreed documentation, warning, or backend change prevents silent empty results and is covered by the relevant verification.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java, mysql, redis
Domain
backend, databases, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.