infiniflow / infiniflow/infinity
[Bug]: many-term match_text aborts the server: number of bits is unsupported in DecodeCurrentTFBuffer
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 445
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 7
Description
### Is there an existing issue for the same bug?
- [x] I have checked the existing issues.
### Version or Commit ID
v0.7.3 (d755c5a), also present on main (eca726625)
### Other environment information
- Hardware parameters: x86_64 (release tarball `infinity-x86_64-v3`), also reproduced on arm64 (`infiniflow/infinity:v0.7.3-arm64`)
- OS type: Linux, systemd unit and docker
- Others: client is RAGFlow 0.27.1 (`match_text`), fulltext index with `rag-*` analyzers; also reproduces with `standard`
### Actual behavior and How to reproduce it
A fulltext `match_text` query with many terms (a whole email body, 200+ terms) kills the server:
```
[critical] Error: TerminateHandler: Unhandled Exception: number of bits is unsupported
[critical] Error: Aborted
```
sometimes as
```
[critical] Error: Decode posting FAILEDF@src/storage/common/int_encoder.cppm:109
```
Stack (identical on x86_64 and arm64):
```
FastPForLib::simdunpack
FastPForLib::SIMDBinaryPacking::decodeArray
infinity::IntEncoder >::Decode int_encoder.cppm:113
infinity::SkipIndexDecoder::DecodeCurrentTFBuffer index_decoder.cppm:95
infinity::MultiPostingDecoder::DecodeCurrentTFBuffer multi_posting_decoder_impl.cpp:75
infinity::PostingIterator::DecodeTFBuffer posting_iterator.cppm:101
infinity::TermDocIterator::BM25Score term_doc_iterator_impl.cpp:148
infinity::BlockMaxWandIterator::Next blockmax_wand_iterator_impl.cpp:433
infinity::ExecuteFTSearch
infinity::PhysicalMatch::ExecuteInner
infinity::FragmentTask::OnExecute
```
Steps (fresh instance, default config):
1. Create a table with a varchar column and a fulltext index on it.
2. Insert 6 batches of 2000 docs (40 to 120 words each from a 300-word vocabulary), call `dump_index` after each batch.
3. Run `match_text` on that column with 200 random vocabulary terms, `topn=10`, `{"operator": "or"}`.
The server aborts within seconds, with a single client thread and no concurrent writes. The same load with 12-term queries runs 250k queries without a crash. It also aborts with a single posting chunk, just less reliably.
### Expected behavior
The query returns results, or at worst fails with an error. A query must not terminate the server.
### Additional information
Root cause, from reading v0.7.3:
- `BlockMaxWandIterator::Next` only partially sorts `sorted_iterators_` when there are more than `SORT_SKIP_THRESHOLD` (15) terms. When the pivot lands in the unsorted tail, iterators in `[0, pivot]` can sit on a doc earlier than the pivot doc. BMW calls `NextShallow(d)` on them, which moves the posting block cursor without decoding it, and then `BM25Score()` on them without a `Next(d)`.
- `PostingIterator::GetCurrentTF` then decodes the tf buffer from the byte-slice reader's current position, which is stale. When the shallow skip crossed into another chunk the reader was just reopened at offset 0 of that chunk, so the chunk header is fed to FastPFor. The only guard is `assert(!need_decode_doc_id_)` in `MultiPostingDecoder::DecodeCurrentTFBuffer`, compiled out in release builds.
- `FragmentTask::OnExecute` catches only Infinity's own exception types, so the `std::logic_error` from FastPFor terminates the process.
Fix in a PR to follow.
Contributor guide
Research direction
Reproduce the crash with the documented full-text setup and 200-term match_text query, then read BlockMaxWandIterator::Next and the posting decoder path in index_decoder.cppm, multi_posting_decoder_impl.cpp, posting_iterator.cppm, and int_encoder.cppm. Trace how shallow skipping affects decoding before changing anything. Done means the query returns results or a handled error without terminating the server.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100