antonmedv / antonmedv/fx

Add fast-path for simple filters

Ouverte
#371 7 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
feature
Langage dominant
Go
Étoiles
20.6k
Forks
488
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

I have a ~600MB NDJSON file with ~1M lines (happy to share if it's helpful).

Using `jq` to search for an individual entry:

```
$ time jq 'select(.id==3260839)' file.ndjson
# ~9 seconds
```

Using `fx -s` to do the same is ~4x slower:

```
$ time fx -s file.ndjson '?.id == 3260839'
# ~33s
```

Using `skip` (ala #292) without `-s` is considerably faster but still slower than `jq`:

```
$ time fx file.ndjson 'x.id == 3260839 ? x : skip'
# ~18s
```

Of course, I can `grep` first to speed things up, but it would be nice if this weren't necessary:

```
$ time grep 3260839 file.ndjson | fx -s '?.id == 3260839'
# ~7s
```

Since there's only ~1M rows in this ndjson file, I doubt that running `x => x.id == 3260839` is the bottleneck. Maybe it's just the JSON parsing?

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.