infiniflow / infiniflow/infinity

Bulk ingestion into a single table: DELETE-vs-DELETE txn conflict livelock, compaction stuck replaying the same ts, and session exhaustion (TOO_MANY_CONNECTIONS)

Open
#3,418 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
4.7k
Forks
445
Avg merge
2d 2h
Merged PRs (30d)
7

Description

### Environment

- Infinity `v0.7.0-dev5` (Docker, Kubernetes StatefulSet, limits 4 CPU / 24 GiB). We also reviewed every commit up to `v0.7.3` — none appear to address the behaviors below.
- Client: RAGFlow task executors — 3 pods × 4 insert threads (12 concurrent writers), Python `infinity-sdk==0.7.0` (thrift).
- Workload: bulk ingestion of 962 HTML documents (~140 chunks each, 1024-dim vectors, insert batches of 128 rows) into **one single table**. Note RAGFlow emulates upsert as `DELETE id IN (...)` + `INSERT` on every batch, so concurrent inserts imply concurrent deletes on the same table.

### What happened (three interacting pathologies)

**1. Transaction-conflict storm between concurrent DELETEs (livelock).**
Server log flooded with hundreds of lines per second like:

```
Transaction: 12779305 is conflicted, detailed info: NewTxn conflict reason:
Delete: database: default_db, db_id: 1, table: ragflow_..._..., table_id: 13, deleted: 5228
vs. Delete: database: default_db, db_id: 1, table: ragflow_..._..., table_id: 13, deleted: 5228.
@src/main/query_context_impl.cpp:603
```

Both conflicting transactions are aborted, both clients retry, conflict again — a livelock. The server burned all 4 CPUs on aborted transactions and effective write throughput dropped to ~zero.

**2. Compaction stuck replaying the same ts.**
Under the same load, the log filled with the *identical* line for ~2 hours:

```
[info] Compact begin ts: 11279323
[info] Compact begin ts: 11279323
... (same ts, repeated continuously)
```

Memory grew from 8.5 GiB to 14.8 GiB during this phase. Inserts/reads timed out while it looped. The loop appeared to persist even after writers were scaled down (short observation window); only a pod restart cleared it.

**3. Session exhaustion → `TOO_MANY_CONNECTIONS (5003)`.**
After hours of client read-timeouts and reconnects, even fresh inserts failed with:

```
infinity.common.InfinityException: (, 'Try 10 times, but still failed')
```

Sessions abandoned by timed-out clients do not seem to be reaped, so the limit fills up over time. Only a restart recovered.

### Analysis / feature asks

1. **Write path concurrency model.** Optimistic concurrency + abort-on-conflict makes conflicts the *nominal* case under bulk ingestion into one table. A per-table write queue / group commit (or pessimistic write locks) would turn the livelock into orderly queueing.
2. **Native UPSERT.** RAGFlow's `DELETE + INSERT` per batch is what makes concurrent deletes ubiquitous. An atomic upsert would remove the biggest conflict source at the root.
3. **Compaction backpressure.** Compaction seems to restart from scratch when it conflicts with writers (same `ts` replayed forever) instead of backing off or proceeding incrementally — and in our case it apparently wedged permanently.
4. **Idle/broken session reaper.** Sessions left behind by clients that hit a socket timeout should expire server-side; today they accumulate until `TOO_MANY_CONNECTIONS`.

### Workaround we deployed (client side, in case it helps other RAGFlow users)

- A distributed per-table write semaphore (max 2 concurrent writers per table, cluster-wide) around the `DELETE+INSERT` pair;
- bounded, jittered retries on `conflict`-class errors (an aborted txn wrote nothing, so replays are safe);
- larger insert batches (512 rows) to reduce segment fragmentation.

This restored orderly ingestion at the engine's natural throughput. Happy to provide fuller logs, timings, or run patched builds to help reproduce.

Contributor guide

Open the contributing guide

Research direction

Start at the conflict report in src/main/query_context_impl.cpp:603 and reproduce the concurrent DELETE+INSERT workload described for one table. Track transaction-conflict logs, repeated compaction timestamps, memory growth, and TOO_MANY_CONNECTIONS during retries and reconnects. Done means the workload no longer livelocks, compaction no longer replays one timestamp indefinitely, and abandoned sessions do not exhaust the connection limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.