InternLM / InternLM/HuixiangDou
Cannot decide self.index type
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 179
- PR merge metrics
- No merged PRs in 30d
Description
## error log | 日志或报错信息 | ログ
```
08/12/2024 15:30:44 - [INFO] -faiss.loader->>> Loading faiss with AVX2 support.
08/12/2024 15:30:44 - [INFO] -faiss.loader->>> Could not load library with AVX2 support due to:
ModuleNotFoundError("No module named 'faiss.swigfaiss_avx2'")
08/12/2024 15:30:44 - [INFO] -faiss.loader->>> Loading faiss.
08/12/2024 15:30:44 - [INFO] -faiss.loader->>> Successfully loaded faiss.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3369/3369 [01:12<00:00, 46.57it/s]
2024-08-12 15:31:56.763 | INFO | huixiangdou.primitive.file_operation:summarize:143 - 累计555文件,成功438个,跳过0个,异常117个
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/root/HuixiangDou/huixiangdou/service/feature_store.py", line 384, in
retriever = cache.get(config_path=args.config_path, work_dir=args.work_dir)
File "/root/HuixiangDou/huixiangdou/service/retriever.py", line 273, in get
retriever = Retriever(config_path=config_path,
File "/root/HuixiangDou/huixiangdou/service/retriever.py", line 48, in __init__
self.faiss = Faiss.load_local(dense_path)
File "/root/HuixiangDou/huixiangdou/primitive/faiss.py", line 176, in load_local
raise ValueError('Cannot decide self.index type {}, open https://github.com/InternLM/HuixiangDou/issues/346 for hotfix'.format(type(index)))
ValueError: Cannot decide self.index type , open https://github.com/InternLM/HuixiangDou/issues/346 for hotfix
```
```
Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss
>>> index = faiss.read_index('embedding.faiss')
>>> print(type(index))
```
## context | 编译/运行环境 | バックグラウンド
```
microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Python 3.10.12
```
## how to reproduce | 复现步骤 | 再現方法
1.
2.
3.
## more | 其他 | その他
[embedding.faiss.zip](https://github.com/user-attachments/files/16578745/embedding.faiss.zip)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in huixiangdou/primitive/faiss.py at Faiss.load_local, then trace its caller in huixiangdou/service/retriever.py and the entry point in huixiangdou/service/feature_store.py. Reproduce with the supplied embedding.faiss file and inspect the faiss.read_index result; done means the reported IndexFlat type is handled without raising the ValueError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100