allenai / allenai/ir_datasets

TrecDocs: .Z and .z files are different.

Ouverte
#189 7 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
bug
Langage dominant
Python
Étoiles
391
Forks
58
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

**Describe the bug**
I've stumbled on this before, and it seems like the same issue happens here. `.z` and `.Z` files are not always equivalent, but `TrecDocs` treat them like so by calling `.lower()` on the suffix of the `Path` object:

https://github.com/allenai/ir_datasets/blob/27317b2951a2c7f843ffc7c8d0b245acdc784c7f/ir_datasets/formats/trec.py#L127-L137

.Z files are created by calling the Unix command [compress](https://linux.die.net/man/1/compress):
(from the man page:
> Compress reduces the size of the named files using adaptive Lempel-Ziv coding. Whenever possible, each file is replaced by one with the extension .Z (...)

while .z files are created by using [gzip](https://linux.die.net/man/1/gzip):
> gunzip takes a list of files on its command line and replaces each file whose name ends with .gz, -gz, .z, -z, _z or .Z (...)

Note that gunzip can decompress BOTH formats, in theory, but, it seems like [unlzw3](https://github.com/scivision/unlzw3) can only read the first (.Z)

There are some Disks45 distributions (mine, for instance) that are compressed with `.z` (i.e. using gunzip with option `-S .z`):
> -S .suf --suffix .suf
When compressing, use suffix .suf instead of .gz. Any non-empty suffix can be given, but suffixes other than .z and .gz should be avoided to avoid confusion when files are transferred to other systems.

**Affected dataset(s)**

All that used `TrecDocs`, but Disks45 more likely.

**To Reproduce**
Trying to read documents with a `.z` compressed files results in this:

```python
TypeError: string argument without an encoding
```

**Additional context**
Error is trigged on this line:
https://github.com/allenai/ir_datasets/blob/27317b2951a2c7f843ffc7c8d0b245acdc784c7f/ir_datasets/formats/trec.py#L136

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.