facebookexperimental / facebookexperimental/semcode

FTS index build deadlocks on hosts with few CPUs

オープン
#61 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Rust
スター
171
フォーク
40
平均マージ
4時間 24分
マージ済み PR(30日)
24

説明

`semcode-index --lore ` hangs partway through the FTS index phase on a 3-core machine. The process stays alive with every thread parked: no CPU time accumulates, `/proc//io` stops advancing, and the partially written index file never grows. It does not recover.

`eu-stack` puts one lance CPU-pool thread here:

```
lance_index::scalar::inverted::builder::InnerBuilder::write_posting_lists
-> lance_core::utils::tokio::spawn_cpu
-> async_channel::SendInner::wait
```

Every other thread, including the main thread and the tokio and rayon workers, sits in a futex or epoll wait.

## Mechanism

`write_posting_lists` hands its producer to lance's global CPU pool through `spawn_cpu`, then drains a bounded channel from the caller's runtime. The producer fills the channel and parks in `tx.send_blocking()`, still holding its pool thread. The consumer's `writer.write_record_batch().await` reaches `PrimitiveStructuralEncoder::do_flush`, which calls `spawn_cpu` itself and queues for a pool thread. The pool has none free, because the producers hold them all.

Neither side can advance. For the build to make progress the pool has to keep at least one thread above the number of concurrent producers.

## Why small hosts reach it

lance sizes the pool at `num_cpus` minus `LANCE_IO_CORE_RESERVATION`, which defaults to 2, and the FTS shard count at roughly half `num_cpus`, clamped to the pool size. The channel is `async_channel::bounded(LANCE_FTS_WRITE_QUEUE_SIZE)`, and that default is 1, so a producer blocks as soon as it finishes its second batch.

| cores | pool | shards | spare thread |
| --- | --- | --- | --- |
| 3 | 1 | 1 | no |
| 4 | 2 | 2 | no |
| 6 | 4 | 3 | yes |
| 16 | 14 | 8 | yes |

Four cores or fewer leave no spare thread, so a single index build deadlocks on its own. Building several indices concurrently reaches the same state on larger hosts.

A rebuild from an empty database is the worst case: it creates all five lore FTS indices rather than merging into existing ones, so it meets the deadlock on the first index.

## Workaround

Give the pool headroom and cap the shard count:

```
LANCE_CPU_THREADS=2 LANCE_FTS_NUM_SHARDS=1 semcode-index --lore
```

## Versions

lance 4.0.0, lancedb 0.27.2, semcode 0.1.1.

I have local changes that build the indices one at a time and pick both variables from `main()` before the first lance call, sized so the pool always keeps a thread above the shard count. Not posting a patch yet; filing this so the scenario is recorded and you can say which direction you want. Related to #56, which covers the memory side of running on small hosts.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。