InternLM / InternLM/HuixiangDou

Cannot decide self.index type

Open
#350 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
179
PR merge metrics
No merged PRs in 30d

Description

## error log | 日志或报错信息 | ログ
```
08/12/2024 15:30:44 - [INFO] -faiss.loader->>> Loading faiss with AVX2 support.
08/12/2024 15:30:44 - [INFO] -faiss.loader->>> Could not load library with AVX2 support due to:
ModuleNotFoundError("No module named 'faiss.swigfaiss_avx2'")
08/12/2024 15:30:44 - [INFO] -faiss.loader->>> Loading faiss.
08/12/2024 15:30:44 - [INFO] -faiss.loader->>> Successfully loaded faiss.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3369/3369 [01:12<00:00, 46.57it/s]
2024-08-12 15:31:56.763 | INFO | huixiangdou.primitive.file_operation:summarize:143 - 累计555文件,成功438个,跳过0个,异常117个
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/root/HuixiangDou/huixiangdou/service/feature_store.py", line 384, in
retriever = cache.get(config_path=args.config_path, work_dir=args.work_dir)
File "/root/HuixiangDou/huixiangdou/service/retriever.py", line 273, in get
retriever = Retriever(config_path=config_path,
File "/root/HuixiangDou/huixiangdou/service/retriever.py", line 48, in __init__
self.faiss = Faiss.load_local(dense_path)
File "/root/HuixiangDou/huixiangdou/primitive/faiss.py", line 176, in load_local
raise ValueError('Cannot decide self.index type {}, open https://github.com/InternLM/HuixiangDou/issues/346 for hotfix'.format(type(index)))
ValueError: Cannot decide self.index type , open https://github.com/InternLM/HuixiangDou/issues/346 for hotfix
```
```
Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss
>>> index = faiss.read_index('embedding.faiss')
>>> print(type(index))

```
## context | 编译/运行环境 | バックグラウンド
```
microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Python 3.10.12
```
## how to reproduce | 复现步骤 | 再現方法

1.
2.
3.

## more | 其他 | その他
[embedding.faiss.zip](https://github.com/user-attachments/files/16578745/embedding.faiss.zip)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in huixiangdou/primitive/faiss.py at Faiss.load_local, then trace its caller in huixiangdou/service/retriever.py and the entry point in huixiangdou/service/feature_store.py. Reproduce with the supplied embedding.faiss file and inspect the faiss.read_index result; done means the reported IndexFlat type is handled without raising the ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.