AnswerDotAI / AnswerDotAI/byaldi

Unable to add Metadata to index

Open
#15 4 comments 0 reactions 0 assignees View on GitHub
bug ColPali Upstream
Dominant language
Python
Stars
851
Forks
91
PR merge metrics
No merged PRs in 30d

Description

When trying to add metadata to an index, either using a list of metadata dicts or a mapping of uid to metadata dict (shown below), it always produces a key error.

Example:

```
RAG = RAGMultiModalModel.from_pretrained("vidore/colpali", device='cuda:2', verbose=1)

# contains 29 1-page pdfs
files = glob(os.path.join('/dataset_pdf', '*.pdf'))

# generate simple unique ids
uids = list(range(len(files)))

# get the file names
report_ids = [file.split('/')[-1].split('.pdf')[0] for file in files]

metadata = {uids[i]: {'file_name':report_ids[i]} for i in range(len(uids))}

RAG.index(
input_path='dataset_pdf',
index_name='Documents', # index will be saved at index_root/index_name/
doc_ids=uids,
store_collection_with_index=True,
overwrite=True,
metadata=metadata,

)
```

This produces the following error:

```
report_ids = [file.split('/')[-1].split('.pdf')[0] for file in files]
metadata = {uids[i]: {'file_name':report_ids[i]} for i in range(len(uids))}
--> RAG.index(
input_path='dataset_pdf',
index_name='Documents', # index will be saved at index_root/index_name/
doc_ids=uids,
store_collection_with_index=True,
overwrite=True,
metadata=metadata,
)

File ~/miniconda3/envs/rag/lib/python3.9/site-packages/byaldi/RAGModel.py:111, in RAGMultiModalModel.index(self, input_path, index_name, doc_ids, store_collection_with_index, overwrite, metadata)
def index(
self,
input_path: Union[str, Path],
(...)
...
--> current_metadata = metadata[i] if metadata else None
if current_doc_id in self.doc_ids:
raise ValueError(f"Document ID {current_doc_id} already exists in the index")

KeyError: 0
```

Removing metadata solves this problem, however, it should be ok based on the metadata docstring from RAGMultiModalModel

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.