infiniflow / infiniflow/infinity

[Bug]: many-term match_text aborts the server: number of bits is unsupported in DecodeCurrentTFBuffer

Open
#3,495 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
4.7k
Forks
445
Avg merge
2d 2h
Merged PRs (30d)
7

Description

### Is there an existing issue for the same bug?

- [x] I have checked the existing issues.

### Version or Commit ID

v0.7.3 (d755c5a), also present on main (eca726625)

### Other environment information

- Hardware parameters: x86_64 (release tarball `infinity-x86_64-v3`), also reproduced on arm64 (`infiniflow/infinity:v0.7.3-arm64`)
- OS type: Linux, systemd unit and docker
- Others: client is RAGFlow 0.27.1 (`match_text`), fulltext index with `rag-*` analyzers; also reproduces with `standard`

### Actual behavior and How to reproduce it

A fulltext `match_text` query with many terms (a whole email body, 200+ terms) kills the server:

```
[critical] Error: TerminateHandler: Unhandled Exception: number of bits is unsupported
[critical] Error: Aborted
```

sometimes as

```
[critical] Error: Decode posting FAILEDF@src/storage/common/int_encoder.cppm:109
```

Stack (identical on x86_64 and arm64):

```
FastPForLib::simdunpack
FastPForLib::SIMDBinaryPacking::decodeArray
infinity::IntEncoder >::Decode int_encoder.cppm:113
infinity::SkipIndexDecoder::DecodeCurrentTFBuffer index_decoder.cppm:95
infinity::MultiPostingDecoder::DecodeCurrentTFBuffer multi_posting_decoder_impl.cpp:75
infinity::PostingIterator::DecodeTFBuffer posting_iterator.cppm:101
infinity::TermDocIterator::BM25Score term_doc_iterator_impl.cpp:148
infinity::BlockMaxWandIterator::Next blockmax_wand_iterator_impl.cpp:433
infinity::ExecuteFTSearch
infinity::PhysicalMatch::ExecuteInner
infinity::FragmentTask::OnExecute
```

Steps (fresh instance, default config):

1. Create a table with a varchar column and a fulltext index on it.
2. Insert 6 batches of 2000 docs (40 to 120 words each from a 300-word vocabulary), call `dump_index` after each batch.
3. Run `match_text` on that column with 200 random vocabulary terms, `topn=10`, `{"operator": "or"}`.

The server aborts within seconds, with a single client thread and no concurrent writes. The same load with 12-term queries runs 250k queries without a crash. It also aborts with a single posting chunk, just less reliably.

### Expected behavior

The query returns results, or at worst fails with an error. A query must not terminate the server.

### Additional information

Root cause, from reading v0.7.3:

- `BlockMaxWandIterator::Next` only partially sorts `sorted_iterators_` when there are more than `SORT_SKIP_THRESHOLD` (15) terms. When the pivot lands in the unsorted tail, iterators in `[0, pivot]` can sit on a doc earlier than the pivot doc. BMW calls `NextShallow(d)` on them, which moves the posting block cursor without decoding it, and then `BM25Score()` on them without a `Next(d)`.
- `PostingIterator::GetCurrentTF` then decodes the tf buffer from the byte-slice reader's current position, which is stale. When the shallow skip crossed into another chunk the reader was just reopened at offset 0 of that chunk, so the chunk header is fed to FastPFor. The only guard is `assert(!need_decode_doc_id_)` in `MultiPostingDecoder::DecodeCurrentTFBuffer`, compiled out in release builds.
- `FragmentTask::OnExecute` catches only Infinity's own exception types, so the `std::logic_error` from FastPFor terminates the process.

Fix in a PR to follow.

Contributor guide

Open the contributing guide

Research direction

Reproduce the crash with the documented full-text setup and 200-term match_text query, then read BlockMaxWandIterator::Next and the posting decoder path in index_decoder.cppm, multi_posting_decoder_impl.cpp, posting_iterator.cppm, and int_encoder.cppm. Trace how shallow skipping affects decoding before changing anything. Done means the query returns results or a handled error without terminating the server.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.