AnswerDotAI / AnswerDotAI/byaldi
Ordering of input files nondeterministic, which can assign incorrect doc id, metadata
- Dominant language
- Python
- Stars
- 851
- Forks
- 91
- PR merge metrics
- No merged PRs in 30d
Description
`for i, item in enumerate(list(input_path.iterdir())): ` can return the files in all sorts of ordering styles. I'm not even sure what it is on my local, but it isn't lexical. What ordering are you expecting?
As a result, we don't know what order to pass in the doc_ids and metadata. Let's either come up with predictable ordering or map filenames to ids/metadata so it isn't left to mismapping 0...N indices.
I have some bandwidth, so once we have a direction, I can implement it if you'd like.
Snippet:
https://github.com/AnswerDotAI/byaldi/blob/427a859cfac3010c3e51451458506a19e203a8b7/byaldi/colpali.py#L278-L289
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.