AnswerDotAI / AnswerDotAI/RAGatouille
UnicodeEncodeError in Windows
- Dominant language
- Python
- Stars
- 4k
- Forks
- 276
- PR merge metrics
- No merged PRs in 30d
Description
Hi, first of all thanks for this great library. While preparing data for training using `trainer.prepare_training_data()` , I got this error `UnicodeEncodeError: 'charmap' codec can't encode character '\u0131' in position 18: character maps to `.
What I found is that ragatouille library is trying to write data containing the Turkish character 'ı' (dotless i, Unicode '\u0131') using my system's default Windows encoding (cp1252), which doesn't support this character (this is also the case for some other Turkish characters). The problem was in the function `export_training_data` from ./data/training_data_processor.py. Simply adding `encoding="utf-8"` fixed the issue.
./data/training_data_processsor.py, line 265-272:
```
with open(path / "queries.train.colbert.tsv", "w", encoding="utf-8") as f:
for query, idx in self.query_map.items():
query = query.replace("\t", " ").replace("\n", " ")
f.write(f"{idx}\t{query}\n")
with open(path / "corpus.train.colbert.tsv", "w", encoding="utf-8") as f:
for document, idx in self.passage_map.items():
document = document.replace("\t", " ").replace("\n", " ")
f.write(f"{idx}\t{document}\n")
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Open data/training_data_processor.py and inspect export_training_data around lines 265-272, reached through trainer.prepare_training_data(). Reproduce the export with Turkish characters, then verify that both generated training TSV files are written successfully and preserve the Unicode text on Windows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100