utf8编码字节数大于3的字符无法召回
- Dominant language
- C++
- Stars
- 1.8k
- Forks
- 352
- PR merge metrics
- No merged PRs in 30d
Description
用以下数据建库
```
CMD=add^_
id=1^_
query=𣄃的组词^_
^^
CMD=add^_
id=2^_
query=形容北京城^_
^^
CMD=add^_
id=3^_
query=怎么读𠀋这个字^_
^^
```
用“𣄃“,”𠀋“无法召回原文档。查看代码发现是Analyzer的normalize()函数内部会调用
```
int32_t EncodeConverter::utf8ToUtf16(const char *in, int32_t length, uint16_t *out) {
int32_t ret = 0;
const unsigned char *b = reinterpret_cast(in);
const unsigned char *e = b + length;
bool error = false;
while (b < e) {
if (*b < 128) {
out[ret++] = *b++;
} else if ((*b & 0xE0) == 0xC0) {
uint16_t u = ((*b++) & (0x1F));
if (b < e && (*b & 0xC0) == 0x80) {
out[ret++] = (u << 6) | ((*b++) & 0x3F);
} else {
error = true;
}
} else if ((*b & 0xF0) == 0xE0) {
uint16_t u = ((*b++) & (0xF));
if (b < e && (*b & 0xC0) == 0x80) {
u = (u << 6) | ((*b++) & 0x3F);
if (b < e && (*b & 0xC0) == 0x80) {
out[ret++] = (u << 6) | ((*b++) & 0x3F);
} else {
error = true;
}
} else {
error = true;
}
} else {
error = true;
b++;
}
}
if (error) {
AUTIL_LOG(DEBUG, "invalid utf8 [%s]", std::string(in, length).c_str());
}
return ret;
}
```
这个函数并没有处理utf8字符为四字节的情况,而且在后续的NormalizeTable中也只分配了0x10000个uint16_t的空间,因此只能处理Unicode码小于0x10000的字符。
如果文档包含Unicode码大于0x10000(utf8编码字节数大于3)的字符,需要把analyzer.json里面的normalize_options都设置为true,让代码不走normalize逻辑,但是大小写、全半角、繁简体的转换就得预先处理了。
```
"normalize_options" :
{
"case_sensitive" : false,
"traditional_sensitive" : true,
"width_sensitive" : false
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at Analyzer::normalize() and inspect EncodeConverter::utf8ToUtf16(), which currently handles only UTF-8 sequences up to three bytes. Then trace the NormalizeTable allocation and analyzer.json normalize_options; done means characters above U+10000 can be recalled without requiring all normalization options to be enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100