facebookexperimental / facebookexperimental/semcode

FTS index build deadlocks on hosts with few CPUs

Ouverte
#61 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Rust
Étoiles
171
Forks
40
Merge moyen
4 h 24 min
PR mergées (30 j)
24

Description

`semcode-index --lore ` hangs partway through the FTS index phase on a 3-core machine. The process stays alive with every thread parked: no CPU time accumulates, `/proc//io` stops advancing, and the partially written index file never grows. It does not recover.

`eu-stack` puts one lance CPU-pool thread here:

```
lance_index::scalar::inverted::builder::InnerBuilder::write_posting_lists
-> lance_core::utils::tokio::spawn_cpu
-> async_channel::SendInner::wait
```

Every other thread, including the main thread and the tokio and rayon workers, sits in a futex or epoll wait.

## Mechanism

`write_posting_lists` hands its producer to lance's global CPU pool through `spawn_cpu`, then drains a bounded channel from the caller's runtime. The producer fills the channel and parks in `tx.send_blocking()`, still holding its pool thread. The consumer's `writer.write_record_batch().await` reaches `PrimitiveStructuralEncoder::do_flush`, which calls `spawn_cpu` itself and queues for a pool thread. The pool has none free, because the producers hold them all.

Neither side can advance. For the build to make progress the pool has to keep at least one thread above the number of concurrent producers.

## Why small hosts reach it

lance sizes the pool at `num_cpus` minus `LANCE_IO_CORE_RESERVATION`, which defaults to 2, and the FTS shard count at roughly half `num_cpus`, clamped to the pool size. The channel is `async_channel::bounded(LANCE_FTS_WRITE_QUEUE_SIZE)`, and that default is 1, so a producer blocks as soon as it finishes its second batch.

| cores | pool | shards | spare thread |
| --- | --- | --- | --- |
| 3 | 1 | 1 | no |
| 4 | 2 | 2 | no |
| 6 | 4 | 3 | yes |
| 16 | 14 | 8 | yes |

Four cores or fewer leave no spare thread, so a single index build deadlocks on its own. Building several indices concurrently reaches the same state on larger hosts.

A rebuild from an empty database is the worst case: it creates all five lore FTS indices rather than merging into existing ones, so it meets the deadlock on the first index.

## Workaround

Give the pool headroom and cap the shard count:

```
LANCE_CPU_THREADS=2 LANCE_FTS_NUM_SHARDS=1 semcode-index --lore
```

## Versions

lance 4.0.0, lancedb 0.27.2, semcode 0.1.1.

I have local changes that build the indices one at a time and pick both variables from `main()` before the first lance call, sized so the pool always keeps a thread above the shard count. Not posting a patch yet; filing this so the scenario is recorded and you can say which direction you want. Related to #56, which covers the memory side of running on small hosts.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.