facebook / facebook/zstd

zdict legacy trainer: ZDICT_tryMerge spends its whole quadratic term proving that nothing matches

Abierto
#4,725 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
27.9k
Forks
2.6k
Merge medio
1 d 3 h
PR fusionados (30 d)
8

Descripción

## Summary

`ZDICT_tryMerge` (`lib/dictBuilder/zdict.c:356-416`) walks the whole dictItem
table twice on every call, and `ZDICT_insertDictItem` calls it once per
candidate plus once per round of its merge fixpoint. With
`dictListSize = MAX(MAX(DICTLISTSIZE_DEFAULT, nbSamples), maxDictSize/16)`
(`zdict.c:987`) that table holds at least 10,000 entries, so the pass is the
dominant term of the pattern-finding phase.

Both loops `return` at the first match:

```c
/* tail overlap */
U32 u; for (u=1; u elt.pos) && (table[u].pos <= eltEnd)) {
...
return u;
} }

/* front overlap */
for (u=1; u= elt.pos && table[u].pos < elt.pos` | 250,705,601 | 8.63% |
| loop increment | 178,224,924 | 6.13% |
| `u==eltNbToSkip` | 89,090,754 | 3.07% |

`MEM_read64` is issued 89,068,550 times while `isIncluded` is reached 1,003
times: the content test almost never matches, and each attempt is a random read
into a sample buffer that may be up to `ZDICT_MAX_SAMPLES_SIZE` = 2000 MB.

## Structure available to exploit

Each `dictItem` is an interval `[pos, pos + length)`, and `ZDICT_insertDictItem`
iterates `ZDICT_tryMerge` to a fixpoint, which fuses intersecting intervals into
connected components. The surviving intervals are pairwise disjoint. That gives
three facts:

- a byte is the start of at most one dictItem, so the tail predicate is
answerable from a per-byte count of starts;
- a byte is spanned by at most one dictItem, so the front positional predicate
is answerable from a per-byte count of spans, and when that count is 1 the
predicate has a **single solution**, making the loop's result independent of
the direction it is walked;
- the content predicate is keyed on a 64-bit value, so a counting filter over
hashed prefixes answers it.

Overlap depth being bounded is what allows nibble-width counters to track
coverage exactly, at one byte per sample byte, the same footprint as
`doneMarks`.

## Measured effect of acting on this

Six corpora, Callgrind, `-O2 -fno-inline -g`, gcc 15.2.0 on Ubuntu 26.04 LTS,
against `dev` plus the fix in #4723:

| corpus | bytes | whole run | `ZDICT_tryMerge` |
|---|---|---|---|
| zstd `lib/*.c` | 2,375,034 | 1.751x | 3.429x |
| zstd `lib/*.h` | 858,183 | 2.172x | 3.557x |
| zstd `programs/*` | 468,328 | 1.658x | 3.788x |
| zstd `tests/` + `doc/` | 1,207,601 | 1.651x | 3.831x |
| python sources | 2,067,845 | 2.657x | 3.904x |
| rust sources | 2,460,916 | 1.386x | 3.562x |

Every emitted dictionary is byte-identical to the unmodified trainer on all six
corpora.

## Limits

The whole-run figure varies because the merge phase is a variable share of
training. On the rust corpus `ZDICT_analyzePos` and `ZDICT_count` are 61.7% of
the run, so removing `ZDICT_tryMerge` entirely would still cap that corpus at
1.73x; that phase is a different algorithm and is untouched here. Only
`ZDICT_trainFromBuffer_legacy` and the CLI's `--train-legacy` reach this code;
`ZDICT_trainFromBuffer` routes to `ZDICT_optimizeTrainFromBuffer_fastCover`
(`zdict.c:1111-1127`) and is unaffected. All measurements come from a single
machine, and no 32-bit target was exercised.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.