semcode-index --lore <list> hangs partway through the FTS index phase on a 3-core machine. The process stays alive with every thread parked: no CPU time accumulates, /proc/<pid>/io stops advancing, and the partially written index file never grows. It does not recover.
eu-stack puts one lance CPU-pool thread here:
lance_index::scalar::inverted::builder::InnerBuilder::write_posting_lists
-> lance_core::utils::tokio::spawn_cpu
-> async_channel::SendInner<RecordBatch>::wait
Every other thread, including the main thread and the tokio and rayon workers, sits in a futex or epoll wait.
Mechanism
write_posting_lists hands its producer to lance's global CPU pool through spawn_cpu, then drains a bounded channel from the caller's runtime. The producer fills the channel and parks in tx.send_blocking(), still holding its pool thread. The consumer's writer.write_record_batch().await reaches PrimitiveStructuralEncoder::do_flush, which calls spawn_cpu itself and queues for a pool thread. The pool has none free, because the producers hold them all.
Neither side can advance. For the build to make progress the pool has to keep at least one thread above the number of concurrent producers.
Why small hosts reach it
lance sizes the pool at num_cpus minus LANCE_IO_CORE_RESERVATION, which defaults to 2, and the FTS shard count at roughly half num_cpus, clamped to the pool size. The channel is async_channel::bounded(LANCE_FTS_WRITE_QUEUE_SIZE), and that default is 1, so a producer blocks as soon as it finishes its second batch.
| cores |
pool |
shards |
spare thread |
| 3 |
1 |
1 |
no |
| 4 |
2 |
2 |
no |
| 6 |
4 |
3 |
yes |
| 16 |
14 |
8 |
yes |
Four cores or fewer leave no spare thread, so a single index build deadlocks on its own. Building several indices concurrently reaches the same state on larger hosts.
A rebuild from an empty database is the worst case: it creates all five lore FTS indices rather than merging into existing ones, so it meets the deadlock on the first index.
Workaround
Give the pool headroom and cap the shard count:
LANCE_CPU_THREADS=2 LANCE_FTS_NUM_SHARDS=1 semcode-index --lore <list>
Versions
lance 4.0.0, lancedb 0.27.2, semcode 0.1.1.
I have local changes that build the indices one at a time and pick both variables from main() before the first lance call, sized so the pool always keeps a thread above the shard count. Not posting a patch yet; filing this so the scenario is recorded and you can say which direction you want. Related to #56, which covers the memory side of running on small hosts.
semcode-index --lore <list>hangs partway through the FTS index phase on a 3-core machine. The process stays alive with every thread parked: no CPU time accumulates,/proc/<pid>/iostops advancing, and the partially written index file never grows. It does not recover.eu-stackputs one lance CPU-pool thread here:Every other thread, including the main thread and the tokio and rayon workers, sits in a futex or epoll wait.
Mechanism
write_posting_listshands its producer to lance's global CPU pool throughspawn_cpu, then drains a bounded channel from the caller's runtime. The producer fills the channel and parks intx.send_blocking(), still holding its pool thread. The consumer'swriter.write_record_batch().awaitreachesPrimitiveStructuralEncoder::do_flush, which callsspawn_cpuitself and queues for a pool thread. The pool has none free, because the producers hold them all.Neither side can advance. For the build to make progress the pool has to keep at least one thread above the number of concurrent producers.
Why small hosts reach it
lance sizes the pool at
num_cpusminusLANCE_IO_CORE_RESERVATION, which defaults to 2, and the FTS shard count at roughly halfnum_cpus, clamped to the pool size. The channel isasync_channel::bounded(LANCE_FTS_WRITE_QUEUE_SIZE), and that default is 1, so a producer blocks as soon as it finishes its second batch.Four cores or fewer leave no spare thread, so a single index build deadlocks on its own. Building several indices concurrently reaches the same state on larger hosts.
A rebuild from an empty database is the worst case: it creates all five lore FTS indices rather than merging into existing ones, so it meets the deadlock on the first index.
Workaround
Give the pool headroom and cap the shard count:
Versions
lance 4.0.0, lancedb 0.27.2, semcode 0.1.1.
I have local changes that build the indices one at a time and pick both variables from
main()before the first lance call, sized so the pool always keeps a thread above the shard count. Not posting a patch yet; filing this so the scenario is recorded and you can say which direction you want. Related to #56, which covers the memory side of running on small hosts.