facebook / facebook/zstd

Possible Search Improvement

Ouverte
#2,587 3 commentaires 3 réactions 0 personnes assignées Voir sur GitHub
optimization
Langage dominant
C
Étoiles
27.9k
Forks
2.6k
Merge moyen
1 j 3 h
PR mergées (30 j)
8

Description

**Is your feature request related to a problem? Please describe.**

In my [Go Zstd Compressor](https://github.com/klauspost/compress/tree/master/zstd#zstd) I have found an [alternate search](https://github.com/klauspost/compress/pull/364) method that gives good improvements in cases where multiple candidates are searched and checked for match length.

In short the strategy is to use the current search strategy and when there is a match calculate a hash starting at input[match_start + match_length]. This will give a hash that can potentially continue the current match.

To check if the new match is better, the (offset_at_hash - match_len) is simply checked and scored as a regular candidate. In the linked code I also check the previous match offset and I've found that to also give a good improvement, mainly in machine generated data (JSON, CSV, marshalled data).

I've found that even with many candidates checked this can still yield good improvements that are worth the cost. I don't know if this has been tried+rejected, but I thought I would forward my findings.

**Describe the solution you'd like**

This could possibly fit in the [lazy matcher](https://github.com/facebook/zstd/blob/dev/lib/compress/zstd_lazy.c#L359), where it would take the best found match and check if the end-of-match can give a better result.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.