AnswerDotAI / AnswerDotAI/byaldi

Ordering of input files nondeterministic, which can assign incorrect doc id, metadata

Open
#3 2 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
851
Forks
91
PR merge metrics
No merged PRs in 30d

Description

`for i, item in enumerate(list(input_path.iterdir())): ` can return the files in all sorts of ordering styles. I'm not even sure what it is on my local, but it isn't lexical. What ordering are you expecting?

As a result, we don't know what order to pass in the doc_ids and metadata. Let's either come up with predictable ordering or map filenames to ids/metadata so it isn't left to mismapping 0...N indices.

I have some bandwidth, so once we have a direction, I can implement it if you'd like.

Snippet:

https://github.com/AnswerDotAI/byaldi/blob/427a859cfac3010c3e51451458506a19e203a8b7/byaldi/colpali.py#L278-L289

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.