allenai / allenai/ir_datasets

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 3417: character maps to <undefined> when trying to decode docs

Aperta
#208 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug
Lingua principale
Python
Stelle
391
Fork
58
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

**Describe the bug**
The library was unable to decode byte into character.

**Affected dataset(s)**
- `msmarco-passage/dev/small`

**To Reproduce**
Steps to reproduce the behavior:
1. Make sure `collectionandqueries.tar.gz` has already been downloaded in the respective dataset folder in `~/.ir_datasets` folder
2. Run:
```
import ir_datasets
train = ir_datasets.load('msmarco-passage/dev/small')
for doc in train.docs_iter():
doc
```
3. Wait for it to run, and you will see an error:
```
[INFO] [starting] fixing encoding
[INFO] [finished] fixing encoding: [07:07] [3.06GB] [7.16MB/s]

---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
d:\Repos\XpressAI\vecto-reranking\1 - Dataset Exploration.ipynb Cell 6 in ()
----> [1](vscode-notebook-cell:/d%3A/Repos/XpressAI/vecto-reranking/1%20-%20Dataset%20Exploration.ipynb#W5sZmlsZQ%3D%3D?line=0) for doc in train.docs_iter():
[2](vscode-notebook-cell:/d%3A/Repos/XpressAI/vecto-reranking/1%20-%20Dataset%20Exploration.ipynb#W5sZmlsZQ%3D%3D?line=1) doc

File d:\Repos\XpressAI\vecto-reranking\venv\lib\site-packages\ir_datasets\util\__init__.py:147, in DocstoreSplitter.__next__(self)
146 def __next__(self):
--> 147 return next(self.it)

File d:\Repos\XpressAI\vecto-reranking\venv\lib\site-packages\ir_datasets\formats\tsv.py:92, in TsvIter.__next__(self)
91 def __next__(self):
---> 92 line = next(self.line_iter)
93 cols = line.rstrip('\n').split('\t')
94 num_cols = len(self.cls._fields)

File d:\Repos\XpressAI\vecto-reranking\venv\lib\site-packages\ir_datasets\formats\tsv.py:30, in FileLineIter.__next__(self)
28 self.stream = io.TextIOWrapper(self.ctxt.enter_context(self.dlc.stream()))
29 while self.pos < self.start:
---> 30 line = self.stream.readline()
31 if line != '\n':
32 self.pos += 1

File ~\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py:23, in IncrementalDecoder.decode(self, input, final)
22 def decode(self, input, final=False):
---> 23 return codecs.charmap_decode(input,self.errors,decoding_table)[0]

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 3417: character maps to
```

**Expected behavior**
Decoding completes without error.

**Additional context**
Screenshot:
![image](https://user-images.githubusercontent.com/59016908/188239635-91f67e89-54dd-4880-b0e2-549fc18b7ece.png)

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.