TrecDocs: .Z and .z files are different.
- Langage dominant
- Python
- Étoiles
- 391
- Forks
- 58
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
**Describe the bug**
I've stumbled on this before, and it seems like the same issue happens here. `.z` and `.Z` files are not always equivalent, but `TrecDocs` treat them like so by calling `.lower()` on the suffix of the `Path` object:
https://github.com/allenai/ir_datasets/blob/27317b2951a2c7f843ffc7c8d0b245acdc784c7f/ir_datasets/formats/trec.py#L127-L137
.Z files are created by calling the Unix command [compress](https://linux.die.net/man/1/compress):
(from the man page:
> Compress reduces the size of the named files using adaptive Lempel-Ziv coding. Whenever possible, each file is replaced by one with the extension .Z (...)
while .z files are created by using [gzip](https://linux.die.net/man/1/gzip):
> gunzip takes a list of files on its command line and replaces each file whose name ends with .gz, -gz, .z, -z, _z or .Z (...)
Note that gunzip can decompress BOTH formats, in theory, but, it seems like [unlzw3](https://github.com/scivision/unlzw3) can only read the first (.Z)
There are some Disks45 distributions (mine, for instance) that are compressed with `.z` (i.e. using gunzip with option `-S .z`):
> -S .suf --suffix .suf
When compressing, use suffix .suf instead of .gz. Any non-empty suffix can be given, but suffixes other than .z and .gz should be avoided to avoid confusion when files are transferred to other systems.
**Affected dataset(s)**
All that used `TrecDocs`, but Disks45 more likely.
**To Reproduce**
Trying to read documents with a `.z` compressed files results in this:
```python
TypeError: string argument without an encoding
```
**Additional context**
Error is trigged on this line:
https://github.com/allenai/ir_datasets/blob/27317b2951a2c7f843ffc7c8d0b245acdc784c7f/ir_datasets/formats/trec.py#L136
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.