facebookexperimental / facebookexperimental/semcode

FTS index build deadlocks on hosts with few CPUs

Abierto
#61 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Rust
Estrellas
171
Forks
40
Merge medio
4 h 24 min
PR fusionados (30 d)
24

Descripción

`semcode-index --lore ` hangs partway through the FTS index phase on a 3-core machine. The process stays alive with every thread parked: no CPU time accumulates, `/proc//io` stops advancing, and the partially written index file never grows. It does not recover.

`eu-stack` puts one lance CPU-pool thread here:

```
lance_index::scalar::inverted::builder::InnerBuilder::write_posting_lists
-> lance_core::utils::tokio::spawn_cpu
-> async_channel::SendInner::wait
```

Every other thread, including the main thread and the tokio and rayon workers, sits in a futex or epoll wait.

## Mechanism

`write_posting_lists` hands its producer to lance's global CPU pool through `spawn_cpu`, then drains a bounded channel from the caller's runtime. The producer fills the channel and parks in `tx.send_blocking()`, still holding its pool thread. The consumer's `writer.write_record_batch().await` reaches `PrimitiveStructuralEncoder::do_flush`, which calls `spawn_cpu` itself and queues for a pool thread. The pool has none free, because the producers hold them all.

Neither side can advance. For the build to make progress the pool has to keep at least one thread above the number of concurrent producers.

## Why small hosts reach it

lance sizes the pool at `num_cpus` minus `LANCE_IO_CORE_RESERVATION`, which defaults to 2, and the FTS shard count at roughly half `num_cpus`, clamped to the pool size. The channel is `async_channel::bounded(LANCE_FTS_WRITE_QUEUE_SIZE)`, and that default is 1, so a producer blocks as soon as it finishes its second batch.

| cores | pool | shards | spare thread |
| --- | --- | --- | --- |
| 3 | 1 | 1 | no |
| 4 | 2 | 2 | no |
| 6 | 4 | 3 | yes |
| 16 | 14 | 8 | yes |

Four cores or fewer leave no spare thread, so a single index build deadlocks on its own. Building several indices concurrently reaches the same state on larger hosts.

A rebuild from an empty database is the worst case: it creates all five lore FTS indices rather than merging into existing ones, so it meets the deadlock on the first index.

## Workaround

Give the pool headroom and cap the shard count:

```
LANCE_CPU_THREADS=2 LANCE_FTS_NUM_SHARDS=1 semcode-index --lore
```

## Versions

lance 4.0.0, lancedb 0.27.2, semcode 0.1.1.

I have local changes that build the indices one at a time and pick both variables from `main()` before the first lance call, sized so the pool always keeps a thread above the shard count. Not posting a patch yet; filing this so the scenario is recorded and you can say which direction you want. Related to #56, which covers the memory side of running on small hosts.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.