facebook / facebook/zstd

zdict legacy trainer: ZDICT_tryMerge spends its whole quadratic term proving that nothing matches

Đang mở
#4,725 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
C
Star
27.9k
Fork
2.6k
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
8

Mô tả

## Summary

`ZDICT_tryMerge` (`lib/dictBuilder/zdict.c:356-416`) walks the whole dictItem
table twice on every call, and `ZDICT_insertDictItem` calls it once per
candidate plus once per round of its merge fixpoint. With
`dictListSize = MAX(MAX(DICTLISTSIZE_DEFAULT, nbSamples), maxDictSize/16)`
(`zdict.c:987`) that table holds at least 10,000 entries, so the pass is the
dominant term of the pattern-finding phase.

Both loops `return` at the first match:

```c
/* tail overlap */
U32 u; for (u=1; u elt.pos) && (table[u].pos <= eltEnd)) {
...
return u;
} }

/* front overlap */
for (u=1; u= elt.pos && table[u].pos < elt.pos` | 250,705,601 | 8.63% |
| loop increment | 178,224,924 | 6.13% |
| `u==eltNbToSkip` | 89,090,754 | 3.07% |

`MEM_read64` is issued 89,068,550 times while `isIncluded` is reached 1,003
times: the content test almost never matches, and each attempt is a random read
into a sample buffer that may be up to `ZDICT_MAX_SAMPLES_SIZE` = 2000 MB.

## Structure available to exploit

Each `dictItem` is an interval `[pos, pos + length)`, and `ZDICT_insertDictItem`
iterates `ZDICT_tryMerge` to a fixpoint, which fuses intersecting intervals into
connected components. The surviving intervals are pairwise disjoint. That gives
three facts:

- a byte is the start of at most one dictItem, so the tail predicate is
answerable from a per-byte count of starts;
- a byte is spanned by at most one dictItem, so the front positional predicate
is answerable from a per-byte count of spans, and when that count is 1 the
predicate has a **single solution**, making the loop's result independent of
the direction it is walked;
- the content predicate is keyed on a 64-bit value, so a counting filter over
hashed prefixes answers it.

Overlap depth being bounded is what allows nibble-width counters to track
coverage exactly, at one byte per sample byte, the same footprint as
`doneMarks`.

## Measured effect of acting on this

Six corpora, Callgrind, `-O2 -fno-inline -g`, gcc 15.2.0 on Ubuntu 26.04 LTS,
against `dev` plus the fix in #4723:

| corpus | bytes | whole run | `ZDICT_tryMerge` |
|---|---|---|---|
| zstd `lib/*.c` | 2,375,034 | 1.751x | 3.429x |
| zstd `lib/*.h` | 858,183 | 2.172x | 3.557x |
| zstd `programs/*` | 468,328 | 1.658x | 3.788x |
| zstd `tests/` + `doc/` | 1,207,601 | 1.651x | 3.831x |
| python sources | 2,067,845 | 2.657x | 3.904x |
| rust sources | 2,460,916 | 1.386x | 3.562x |

Every emitted dictionary is byte-identical to the unmodified trainer on all six
corpora.

## Limits

The whole-run figure varies because the merge phase is a variable share of
training. On the rust corpus `ZDICT_analyzePos` and `ZDICT_count` are 61.7% of
the run, so removing `ZDICT_tryMerge` entirely would still cap that corpus at
1.73x; that phase is a different algorithm and is untouched here. Only
`ZDICT_trainFromBuffer_legacy` and the CLI's `--train-legacy` reach this code;
`ZDICT_trainFromBuffer` routes to `ZDICT_optimizeTrainFromBuffer_fastCover`
(`zdict.c:1111-1127`) and is unaffected. All measurements come from a single
machine, and no 32-bit target was exercised.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.