AnswerDotAI / AnswerDotAI/RAGatouille

UnicodeEncodeError in Windows

Open
#268 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4k
Forks
276
PR merge metrics
No merged PRs in 30d

Description

Hi, first of all thanks for this great library. While preparing data for training using `trainer.prepare_training_data()` , I got this error `UnicodeEncodeError: 'charmap' codec can't encode character '\u0131' in position 18: character maps to `.

What I found is that ragatouille library is trying to write data containing the Turkish character 'ı' (dotless i, Unicode '\u0131') using my system's default Windows encoding (cp1252), which doesn't support this character (this is also the case for some other Turkish characters). The problem was in the function `export_training_data` from ./data/training_data_processor.py. Simply adding `encoding="utf-8"` fixed the issue.

./data/training_data_processsor.py, line 265-272:
```
with open(path / "queries.train.colbert.tsv", "w", encoding="utf-8") as f:
for query, idx in self.query_map.items():
query = query.replace("\t", " ").replace("\n", " ")
f.write(f"{idx}\t{query}\n")
with open(path / "corpus.train.colbert.tsv", "w", encoding="utf-8") as f:
for document, idx in self.passage_map.items():
document = document.replace("\t", " ").replace("\n", " ")
f.write(f"{idx}\t{document}\n")
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Open data/training_data_processor.py and inspect export_training_data around lines 265-272, reached through trainer.prepare_training_data(). Reproduce the export with Turkish characters, then verify that both generated training TSV files are written successfully and preserve the Unicode text on Windows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.