conductor-oss / conductor-oss/conductor
Cassandra: conductor-server fails startup with "session is closed" during getAllTaskDefsFromDB
- Dominant language
- Java
- Stars
- 32.2k
- Forks
- 1k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 41
Description
### Symptom
When bringing up the Cassandra-backed docker stack (`docker/docker-compose-cassandra-es7.yaml`), `conductor-server` starts but never becomes healthy. During its boot-time initialization, the Cassandra driver session closes mid-query, causing repeated startup failures:
```
Caused by: com.datastax.driver.core.exceptions.DriverInternalError:
Unexpected exception thrown
at com.netflix.conductor.cassandra.dao.CassandraMetadataDAO.getAllTaskDefsFromDB(CassandraMetadataDAO.java:358)
Caused by: java.lang.IllegalStateException: Could not send request, session is closed
at com.datastax.driver.core.SessionManager.execute(SessionManager.java:701)
at com.netflix.conductor.cassandra.dao.CassandraMetadataDAO.getAllTaskDefsFromDB(CassandraMetadataDAO.java:358)
```
`/health` never returns OK. Even with a 600s wait, the service does not recover.
### Reproduction
```shell
cd conductor
docker compose -f docker/docker-compose-cassandra-es7.yaml build
docker compose -f docker/docker-compose-cassandra-es7.yaml up
# wait — conductor-server logs will show the stack above repeatedly
curl -i http://localhost:8000/health # never returns 200
```
Reproduced on conductor `main` and on the branch I was testing (`feat/webhooks-from-orkes-split`).
### Probable root cause (both likely contribute)
1. **Cqlsh-passes-but-not-ready startup race.** The compose healthcheck on `conductor-cassandra` is `cqlsh -e \"describe keyspaces\"`. That succeeds the moment the native protocol port accepts connections, well before keyspace creation / schema initialization settles. `conductor-server` boots on the assumption Cassandra is ready, opens a driver session, and the underlying connection gets torn down before the first real query lands.
2. **Tight heap settings.** The compose sets `MAX_HEAP_SIZE=512M / HEAP_NEWSIZE=128M` on `cassandra:4`. Borderline on modern JVMs. Under boot load the driver may observe intermittent disconnects from GC pauses.
### Suggested fixes (cheapest first)
1. Bump cassandra heap in the compose to at least `MAX_HEAP_SIZE=1G / HEAP_NEWSIZE=256M`.
2. Tighten the cassandra healthcheck so it only reports healthy after the application keyspace exists — e.g. `cqlsh -e \"SELECT keyspace_name FROM system_schema.keyspaces WHERE keyspace_name='conductor'\"` with generous `retries`.
3. In `CassandraBaseDAO` (or wherever the startup session is opened), add bounded retry around the first query so a single transient disconnect doesn't kill the whole process.
### Impact
Blocks any docker-based Cassandra deployment of conductor-server. Surfaced while validating a separate PR's persistence matrix.
### How I found this
Running a new composite-workflow stress harness against each persistence backend (postgres, mysql, redis, cassandra) for a feature PR. Postgres and Redis passed cleanly; MySQL fails for a separate pre-existing reason (#1104); Cassandra fails as described above. The failing code path (`CassandraMetadataDAO.getAllTaskDefsFromDB`) is pre-existing core conductor code, not introduced by the PR I was testing.
Contributor guide
Research direction
Start with docker/docker-compose-cassandra-es7.yaml and CassandraMetadataDAO.java around line 358, then reproduce with the documented docker compose commands and inspect the Cassandra healthcheck and startup logs. The work is done when the Cassandra-backed conductor-server starts reliably and curl http://localhost:8000/health returns 200 without repeated session-closed failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, docker, docker-compose, java
- Domain
- databases, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100