null value in column node_id crashes indexing pipelines after migration issues

Describe the bug
The problem arise after the problem of "migration 17 was previously applied but missing in resolved migrations". When previous migration problem happened, we tried recover from it by deleting all migrations in _sql_migrations where version>=17. The server then starts successfully. However, while ingesting, the null value problem happens and crashes indexing pipeline. Which crashes other services since the ingest queue will fill up quickly after this and reject all ingestion request.

Wanted to ask if there are any ways to recover from this kind of error and not loosing data.

Also wanted to know the cause of the "migration 17 was previously applied but missing in resolved migrations" problem and how to recover from it.

Logs:

2024-08-27T21:43:46.327Z ERROR index-doc-batches{index_id=cerus-logfile-rawline-v1 source_id=_ingest-api-source pipeline_uid=01J6AVPG88MT5G9WRV3NNP6C4Q workbench_id=01J6AWMW6MK06EVTNKHV7KKDE0}:uploader:stage_splits{split_ids="[\"01J6AWMW959T3G44GZVN5R67E4\"]"}: quickwit_metastore::metastore::postgres::error: postgresql-error error=PgDatabaseError { severity: Error, code: "23502", message: "null value in column \"node_id\" of relation \"splits\" violates not-null constraint", detail: Some("Failing row contains (01J6AWMW959T3G44GZVN5R67E4, Staged, 1724794870, 1724795017, {}, {\"version\":\"0.8\",\"split_id\":\"01J6AWMW959T3G44GZVN5R67E4\",\"index_..., cerus-logfile-rawline-v1:01J4T6W0S1KY145TQ898PEFXFY, 2024-08-27 21:43:46.232142, 2024-08-27 21:43:46.232142, null, 0, 2024-08-29 21:43:46, null)."), hint: None, position: None, where: None, schema: Some("public"), table: Some("splits"), column: Some("node_id"), data_type: None, constraint: None, file: Some("execMain.c"), line: Some(1974), routine: Some("ExecConstraints") }
2024-08-27T21:43:46.327Z  WARN index-doc-batches{index_id=cerus-logfile-rawline-v1 source_id=_ingest-api-source pipeline_uid=01J6AVPG88MT5G9WRV3NNP6C4Q workbench_id=01J6AWMW6MK06EVTNKHV7KKDE0}:uploader:stage_splits{split_ids="[\"01J6AWMW959T3G44GZVN5R67E4\"]"}: quickwit_metastore::metastore::postgres::metastore: rollback
2024-08-27T21:43:46.404Z ERROR quickwit_actors::spawn_builder: actor-failure cause=failed to receive command from uploader

Caused by:
   channel closed exit_status=Failure(failed to receive command from uploader

Caused by:
   channel closed)
2024-08-27T21:43:46.404Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-cold-DmVd exit_status=failure(cause=failed to receive command from uploader

Caused by:
   channel closed)
2024-08-27T21:43:46.404Z ERROR quickwit_actors::actor_context: exit activating-kill-switch actor=quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-cold-DmVd exit_status=Failure(failed to receive command from uploader

Caused by:
   channel closed)
2024-08-27T21:43:46.404Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Publisher-small-JH9u exit_status=killed
2024-08-27T21:43:46.422Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Indexer-misty-y1nk exit_status=killed
2024-08-27T21:43:46.422Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::doc_processor::DocProcessor-late-pbxr exit_status=killed
2024-08-27T21:43:46.422Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=SourceActor-icy-uHQU exit_status=killed
2024-08-27T21:43:46.434Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=quickwit_indexing::actors::index_serializer::IndexSerializer-solitary-CV8N exit_status=killed
2024-08-27T21:43:46.434Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=Packager-black-nUJo exit_status=killed
2024-08-27T21:43:46.435Z  INFO quickwit_actors::spawn_builder: actor-exit actor_id=IndexUploader-cold-zJfk exit_status=killed
2024-08-27T21:43:46.843Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="SourceActor-icy-uHQU"
2024-08-27T21:43:46.843Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="quickwit_indexing::actors::doc_processor::DocProcessor-late-pbxr"
2024-08-27T21:43:46.843Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="Indexer-misty-y1nk"
2024-08-27T21:43:46.843Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="quickwit_indexing::actors::index_serializer::IndexSerializer-solitary-CV8N"
2024-08-27T21:43:46.843Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="Packager-black-nUJo"
2024-08-27T21:43:46.843Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="IndexUploader-cold-zJfk"
2024-08-27T21:43:46.843Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-cold-DmVd"
2024-08-27T21:43:46.843Z ERROR quickwit_actors::actor_handle: actor-exit-without-success actor="Publisher-small-JH9u"
2024-08-27T21:43:46.843Z ERROR quickwit_indexing::actors::indexing_pipeline: Indexing pipeline failure. pipeline_id=IndexingPipelineId { node_id: "prime-etl-01", index_uid: IndexUid { index_id: "cerus-logfile-rawline-v1", incarnation_id: Ulid(2083174441638835175660655156021163518) }, source_id: "_ingest-api-source", pipeline_uid: Pipeline(01J6AVPG88MT5G9WRV3NNP6C4Q) } generation=2 healthy_actors=[] failed_or_unhealthy_actors=["SourceActor-icy-uHQU", "quickwit_indexing::actors::doc_processor::DocProcessor-late-pbxr", "Indexer-misty-y1nk", "quickwit_indexing::actors::index_serializer::IndexSerializer-solitary-CV8N", "Packager-black-nUJo", "IndexUploader-cold-zJfk", "quickwit_indexing::actors::sequencer::Sequencer<quickwit_indexing::actors::publisher::Publisher>-cold-DmVd", "Publisher-small-JH9u"] success_actors=[]

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

1. Run one instance of docker quickwit 0.8.2 and another instance of binary quickwit 0.8.0. restarted each for a few times
2. Either server won't start because of the migration error on postgres
3. Deleted all migrations >= 17 in postgres using psql
4. Start one instance of docker quickwit 0.8.2
5. Ingest data by posting to REST API
6. Index pipeline crashes and ingestion queue fill up and reject all ingestion requests

Expected behavior
The server would reapply necessary migrations and work normally

Configuration:
Please provide:

1. Output of quickwit --version
   0.8.2 docker image and 0.8.0 compiled binary

Start by reproducing the migration mismatch with Quickwit 0.8.0 and 0.8.2, PostgreSQL, and the documented migration-table deletion steps. Trace the migration handling and indexing pipeline around the reported splits.node_id constraint failure. Done should include a defined recovery path that preserves data, reapplies or reconciles migrations safely, and prevents ingestion pipelines from crashing.