alibaba / alibaba/havenask

utf8编码字节数大于3的字符无法召回

Open
#190 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.8k
Forks
352
PR merge metrics
No merged PRs in 30d

Description

用以下数据建库
```
CMD=add^_
id=1^_
query=𣄃的组词^_
^^
CMD=add^_
id=2^_
query=形容北京城^_
^^
CMD=add^_
id=3^_
query=怎么读𠀋这个字^_
^^
```
用“𣄃“,”𠀋“无法召回原文档。查看代码发现是Analyzer的normalize()函数内部会调用
```
int32_t EncodeConverter::utf8ToUtf16(const char *in, int32_t length, uint16_t *out) {
int32_t ret = 0;
const unsigned char *b = reinterpret_cast(in);
const unsigned char *e = b + length;
bool error = false;
while (b < e) {
if (*b < 128) {
out[ret++] = *b++;
} else if ((*b & 0xE0) == 0xC0) {
uint16_t u = ((*b++) & (0x1F));
if (b < e && (*b & 0xC0) == 0x80) {
out[ret++] = (u << 6) | ((*b++) & 0x3F);
} else {
error = true;
}
} else if ((*b & 0xF0) == 0xE0) {
uint16_t u = ((*b++) & (0xF));
if (b < e && (*b & 0xC0) == 0x80) {
u = (u << 6) | ((*b++) & 0x3F);
if (b < e && (*b & 0xC0) == 0x80) {
out[ret++] = (u << 6) | ((*b++) & 0x3F);
} else {
error = true;
}
} else {
error = true;
}
} else {
error = true;
b++;
}
}

if (error) {
AUTIL_LOG(DEBUG, "invalid utf8 [%s]", std::string(in, length).c_str());
}
return ret;
}
```
这个函数并没有处理utf8字符为四字节的情况,而且在后续的NormalizeTable中也只分配了0x10000个uint16_t的空间,因此只能处理Unicode码小于0x10000的字符。

如果文档包含Unicode码大于0x10000(utf8编码字节数大于3)的字符,需要把analyzer.json里面的normalize_options都设置为true,让代码不走normalize逻辑,但是大小写、全半角、繁简体的转换就得预先处理了。
```
"normalize_options" :
{
"case_sensitive" : false,
"traditional_sensitive" : true,
"width_sensitive" : false
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at Analyzer::normalize() and inspect EncodeConverter::utf8ToUtf16(), which currently handles only UTF-8 sequences up to three bytes. Then trace the NormalizeTable allocation and analyzer.json normalize_options; done means characters above U+10000 can be recalled without requiring all normalization options to be enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.