llmware-ai / llmware-ai/llmware

Creating embedding with MongoDB text store when library contains CSV file fails

Open
#475 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
14.8k
Forks
2.9k
PR merge metrics
No merged PRs in 30d

Description

Hi all! I think I may have found a bug related to creating embeddings of CSV files.

**When attempting to create an embedding of a library (with ChromaDB as vector_db), where the library has a CSV file added, I'm getting the following exception:**

```python
File "C:\Files\llmware-main\llmware\embeddings.py", line 2164, in create_new_embedding
text_search = block["text_search"].strip()
^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'strip'
```

**I certainly could be doing something wrong, but on the line mentioned, block["text_search"] is a list of lists representing the rows of the CSV, but the code then attempts to do a strip() on the list as if it was a str, causing the error. Simply avoiding the strip() based on instance type doesn't work as errors are then picked up elsewhere.

**I'm able to reproduce with the latest main branch using the following sample code (with above setup included):**

Setup/Boilerplate Minimized for Simplicity

```python
from llmware.configs import ChromaDBConfig, LLMWareConfig, MongoConfig
from llmware.library import Library

vector_db = "chromadb"
active_db = "mongo"
debug_int = 3
home_dir = "LLMWARE_WORKSPACE"
path = "data"
chroma_path = "ChromaDB\\data"

account_name = "testaccount"
library_name = "testlib"
file_path = "addresses.csv"
embedding_model_name = "industry-bert-contracts"

mongo_username = "mongousr"
mongo_password = "XXXXX"
mongo_dbname = "llmware_libraries"
mongo_db_uri = "mongodb://" + mongo_username + ":" + mongo_password + "@127.0.0.1:27017/"

def setup():

LLMWareConfig().set_vector_db(vector_db)
LLMWareConfig().set_active_db(active_db)
LLMWareConfig().set_config("debug_mode", debug_int)

LLMWareConfig().set_home(home_dir)

LLMWareConfig().set_llmware_path_name(path)
LLMWareConfig().setup_llmware_workspace()

MongoConfig.set_config("user_name", mongo_username)
MongoConfig.set_config("pw", mongo_password)
MongoConfig.set_config("db_name", mongo_dbname)
MongoConfig.set_config("db_uri", mongo_db_uri)

ChromaDBConfig.set_config("persistent_path", chroma_path)
```


```python

def csv_example():

global library_name, account_name, file_path, embedding_model_name, vector_db

library = Library().create_new_library(library_name=library_name, account_name=account_name)

library.add_file(file_path=file_path)

library.install_new_embedding(embedding_model_name=embedding_model_name, vector_db=vector_db)

if __name__ == "__main__":
setup() #
csv_example()

```

**Example addresses.csv file contents (though, it happens with all I've tested):**

```python
a,b,c,d,e,f,g,h
h,g,f,e,d,c,b,a
z,z,z,z,z,z,z,z
a,a,a,a,a,a,a,a
```

Bug or user error? :)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at llmware/embeddings.py line 2164 and reproduce the failure with the setup() and csv_example() workflow using a CSV library and MongoDB/ChromaDB configuration. Trace how the CSV rows reach create_new_embedding and compare that representation with the string expected there. Done means the sample CSV library can install its embedding without the reported list/strip exception or a subsequent representation error.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.