Azure / Azure/azureml-examples

Loading pre-built local Faiss Index causes ValueError (allow_dangerous_deserialization)

Open
#3,058 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Operating System

Windows

### Version Information

Python Version: 3.11.5

```txt
promptflow 1.6.0
promptflow-tools 1.3.0
promptflow_vectordb 0.2.5

langchain 0.1.12
langchain-community 0.0.28
langchain-core 0.1.32
langchain-experimental 0.0.43
langchain-openai 0.0.8
langchain-text-splitters 0.0.1

faiss-cpu 1.7.4
```

### Steps to reproduce

This can be easily reproduced by following the tutorial notebook and then reload the saved Faiss index by rerun the cell

https://github.com/Azure/azureml-examples/blob/df1e23c1fff73421e6cb0753d029488c4c966473/sdk/python/generative-ai/promptflow/create_faiss_index.ipynb#L122-L139

### Expected behavior

Should successfully load pre-built Faiss index without error

### Actual behavior

Got ValueError

https://github.com/langchain-ai/langchain/blob/40f846e65da37a1c00d72da9ea64ebb0f295b016/libs/community/langchain_community/vectorstores/faiss.py#L1054-L1089

### Addition information

Should somehow pass `allow_dangerous_deserialization=True` to use local pickle vector db checkpoint.

I was able to bypass this error by changing `promptflow_vectordb/core/engine/langchain_engine.py`

```py
# From
self.__langchain_faiss = FAISS.load_local(path, LangchainEmbedding(self.__embedding))
# To
self.__langchain_faiss = FAISS.load_local(path, LangchainEmbedding(self.__embedding), allow_dangerous_deserialization=True)
```

Contributor guide

Open the contributing guide

Research direction

Start with promptflow_vectordb/core/engine/langchain_engine.py and the linked create_faiss_index.ipynb reproduction steps. Inspect the FAISS.load_local call and the referenced LangChain behavior, then rerun the notebook's save-and-reload flow; done means a pre-built local index loads successfully without the ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
databases, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.