Azure / Azure/azureml-examples
Loading pre-built local Faiss Index causes ValueError (allow_dangerous_deserialization)
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 1.7k
- Avg merge
- 18h 18m
- Merged PRs (30d)
- 2
Description
### Operating System
Windows
### Version Information
Python Version: 3.11.5
```txt
promptflow 1.6.0
promptflow-tools 1.3.0
promptflow_vectordb 0.2.5
langchain 0.1.12
langchain-community 0.0.28
langchain-core 0.1.32
langchain-experimental 0.0.43
langchain-openai 0.0.8
langchain-text-splitters 0.0.1
faiss-cpu 1.7.4
```
### Steps to reproduce
This can be easily reproduced by following the tutorial notebook and then reload the saved Faiss index by rerun the cell
https://github.com/Azure/azureml-examples/blob/df1e23c1fff73421e6cb0753d029488c4c966473/sdk/python/generative-ai/promptflow/create_faiss_index.ipynb#L122-L139
### Expected behavior
Should successfully load pre-built Faiss index without error
### Actual behavior
Got ValueError
https://github.com/langchain-ai/langchain/blob/40f846e65da37a1c00d72da9ea64ebb0f295b016/libs/community/langchain_community/vectorstores/faiss.py#L1054-L1089
### Addition information
Should somehow pass `allow_dangerous_deserialization=True` to use local pickle vector db checkpoint.
I was able to bypass this error by changing `promptflow_vectordb/core/engine/langchain_engine.py`
```py
# From
self.__langchain_faiss = FAISS.load_local(path, LangchainEmbedding(self.__embedding))
# To
self.__langchain_faiss = FAISS.load_local(path, LangchainEmbedding(self.__embedding), allow_dangerous_deserialization=True)
```
Contributor guide
Research direction
Start with promptflow_vectordb/core/engine/langchain_engine.py and the linked create_faiss_index.ipynb reproduction steps. Inspect the FAISS.load_local call and the referenced LangChain behavior, then rerun the notebook's save-and-reload flow; done means a pre-built local index loads successfully without the ValueError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- databases, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100