zdict legacy trainer: ZDICT_tryMerge spends its whole quadratic term proving that nothing matches
- Langage dominant
- C
- Étoiles
- 27.9k
- Forks
- 2.6k
- Merge moyen
- 1 j 3 h
- PR mergées (30 j)
- 8
Description
## Summary
`ZDICT_tryMerge` (`lib/dictBuilder/zdict.c:356-416`) walks the whole dictItem
table twice on every call, and `ZDICT_insertDictItem` calls it once per
candidate plus once per round of its merge fixpoint. With
`dictListSize = MAX(MAX(DICTLISTSIZE_DEFAULT, nbSamples), maxDictSize/16)`
(`zdict.c:987`) that table holds at least 10,000 entries, so the pass is the
dominant term of the pattern-finding phase.
Both loops `return` at the first match:
```c
/* tail overlap */
U32 u; for (u=1; u elt.pos) && (table[u].pos <= eltEnd)) {
...
return u;
} }
/* front overlap */
for (u=1; u= elt.pos && table[u].pos < elt.pos` | 250,705,601 | 8.63% |
| loop increment | 178,224,924 | 6.13% |
| `u==eltNbToSkip` | 89,090,754 | 3.07% |
`MEM_read64` is issued 89,068,550 times while `isIncluded` is reached 1,003
times: the content test almost never matches, and each attempt is a random read
into a sample buffer that may be up to `ZDICT_MAX_SAMPLES_SIZE` = 2000 MB.
## Structure available to exploit
Each `dictItem` is an interval `[pos, pos + length)`, and `ZDICT_insertDictItem`
iterates `ZDICT_tryMerge` to a fixpoint, which fuses intersecting intervals into
connected components. The surviving intervals are pairwise disjoint. That gives
three facts:
- a byte is the start of at most one dictItem, so the tail predicate is
answerable from a per-byte count of starts;
- a byte is spanned by at most one dictItem, so the front positional predicate
is answerable from a per-byte count of spans, and when that count is 1 the
predicate has a **single solution**, making the loop's result independent of
the direction it is walked;
- the content predicate is keyed on a 64-bit value, so a counting filter over
hashed prefixes answers it.
Overlap depth being bounded is what allows nibble-width counters to track
coverage exactly, at one byte per sample byte, the same footprint as
`doneMarks`.
## Measured effect of acting on this
Six corpora, Callgrind, `-O2 -fno-inline -g`, gcc 15.2.0 on Ubuntu 26.04 LTS,
against `dev` plus the fix in #4723:
| corpus | bytes | whole run | `ZDICT_tryMerge` |
|---|---|---|---|
| zstd `lib/*.c` | 2,375,034 | 1.751x | 3.429x |
| zstd `lib/*.h` | 858,183 | 2.172x | 3.557x |
| zstd `programs/*` | 468,328 | 1.658x | 3.788x |
| zstd `tests/` + `doc/` | 1,207,601 | 1.651x | 3.831x |
| python sources | 2,067,845 | 2.657x | 3.904x |
| rust sources | 2,460,916 | 1.386x | 3.562x |
Every emitted dictionary is byte-identical to the unmodified trainer on all six
corpora.
## Limits
The whole-run figure varies because the merge phase is a variable share of
training. On the rust corpus `ZDICT_analyzePos` and `ZDICT_count` are 61.7% of
the run, so removing `ZDICT_tryMerge` entirely would still cap that corpus at
1.73x; that phase is a different algorithm and is untouched here. Only
`ZDICT_trainFromBuffer_legacy` and the CLI's `--train-legacy` reach this code;
`ZDICT_trainFromBuffer` routes to `ZDICT_optimizeTrainFromBuffer_fastCover`
(`zdict.c:1111-1127`) and is unaffected. All measurements come from a single
machine, and no 32-bit target was exercised.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.