llmware-ai / llmware-ai/llmware
Creating embedding with MongoDB text store when library contains CSV file fails
- Dominant language
- Python
- Stars
- 14.8k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
Hi all! I think I may have found a bug related to creating embeddings of CSV files.
**When attempting to create an embedding of a library (with ChromaDB as vector_db), where the library has a CSV file added, I'm getting the following exception:**
```python
File "C:\Files\llmware-main\llmware\embeddings.py", line 2164, in create_new_embedding
text_search = block["text_search"].strip()
^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'strip'
```
**I certainly could be doing something wrong, but on the line mentioned, block["text_search"] is a list of lists representing the rows of the CSV, but the code then attempts to do a strip() on the list as if it was a str, causing the error. Simply avoiding the strip() based on instance type doesn't work as errors are then picked up elsewhere.
**I'm able to reproduce with the latest main branch using the following sample code (with above setup included):**
Setup/Boilerplate Minimized for Simplicity
```python
from llmware.configs import ChromaDBConfig, LLMWareConfig, MongoConfig
from llmware.library import Library
vector_db = "chromadb"
active_db = "mongo"
debug_int = 3
home_dir = "LLMWARE_WORKSPACE"
path = "data"
chroma_path = "ChromaDB\\data"
account_name = "testaccount"
library_name = "testlib"
file_path = "addresses.csv"
embedding_model_name = "industry-bert-contracts"
mongo_username = "mongousr"
mongo_password = "XXXXX"
mongo_dbname = "llmware_libraries"
mongo_db_uri = "mongodb://" + mongo_username + ":" + mongo_password + "@127.0.0.1:27017/"
def setup():
LLMWareConfig().set_vector_db(vector_db)
LLMWareConfig().set_active_db(active_db)
LLMWareConfig().set_config("debug_mode", debug_int)
LLMWareConfig().set_home(home_dir)
LLMWareConfig().set_llmware_path_name(path)
LLMWareConfig().setup_llmware_workspace()
MongoConfig.set_config("user_name", mongo_username)
MongoConfig.set_config("pw", mongo_password)
MongoConfig.set_config("db_name", mongo_dbname)
MongoConfig.set_config("db_uri", mongo_db_uri)
ChromaDBConfig.set_config("persistent_path", chroma_path)
```
```python
def csv_example():
global library_name, account_name, file_path, embedding_model_name, vector_db
library = Library().create_new_library(library_name=library_name, account_name=account_name)
library.add_file(file_path=file_path)
library.install_new_embedding(embedding_model_name=embedding_model_name, vector_db=vector_db)
if __name__ == "__main__":
setup() #
csv_example()
```
**Example addresses.csv file contents (though, it happens with all I've tested):**
```python
a,b,c,d,e,f,g,h
h,g,f,e,d,c,b,a
z,z,z,z,z,z,z,z
a,a,a,a,a,a,a,a
```
Bug or user error? :)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at llmware/embeddings.py line 2164 and reproduce the failure with the setup() and csv_example() workflow using a CSV library and MongoDB/ChromaDB configuration. Trace how the CSV rows reach create_new_embedding and compare that representation with the string expected there. Done means the sample CSV library can install its embedding without the reported list/strip exception or a subsequent representation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, python
- Domain
- databases, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100