facebook / facebook/zstd

Question: Confirming v1.5.5 ↔ v1.5.7 bidirectional compatibility for ZSTD_compress_usingDict / ZSTD_decompress_usingDict with raw-content dictionaries

Aperta
#4,706 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
27.9k
Fork
2.6k
Merge medio
1g 3h
PR unite (30g)
8

Descrizione

This is a compatibility confirmation question, not a bug — using the bug template because it is the only one available. No decoding failure has been observed.

Prior art in this repository:
- #3110 "ZSTD compatibility" — @Cyan4973 confirmed frame-format stability across versions.
- #3711 "Compatibility between compress/decompress APIs" — confirmed decoders are compatible with frames produced by newer encoders.
- #3802 "Compressing and decompressing with dictionaries, between different zstd versions" — addressed the dictionary case in mixed-version deployments.

A related question was recently filed as #4705 (v1.4.1 ↔ v1.5.7 in a RocksDB context). Our scenario is similar in shape but narrower in version span and specific to the raw-content prefix dictionary API, so filing separately for clarity.

Given those guarantees, we would like a targeted confirmation for our exact API shape before rolling a version bump into a large production deployment.

Scenario

We use zstd for block-level compression in a distributed service, statically linked into each node. We are upgrading from libzstd 1.5.5 to 1.5.7 as a rolling deployment. During the
rolling window (and as a rollback safety net) nodes on either version must be able to decode data produced by the other.

Encoder-side call shape (both versions)

- Single-shot buffer-to-buffer only — no streaming APIs.
- ZSTD_compress_usingDict(cctx, dst, dstCap, src, srcSize, dict, dictSize, level=6)
- ZSTD_decompress_usingDict(dctx, dst, dstCap, src, srcSize, dict, dictSize)
- dict is a raw-content prefix (not a serialized zstd dictionary with the ZSTD_MAGIC_DICTIONARY header); no ZDICT_* calls; no ZSTD_CDict / ZSTD_DDict objects.
- No ZSTD_CCtx_setParameter calls anywhere — every advanced parameter is at library default. Specifically:
- ZSTD_c_format = ZSTD_f_zstd1 (default; standard magic)
- ZSTD_c_checksumFlag = default (off)
- ZSTD_c_contentSizeFlag = default (on; srcSize is passed)
- ZSTD_c_dictIDFlag = default (raw-content prefix ⇒ no dictID embedded)
- ZSTD_c_windowLog, strategy, ZSTD_c_enableLongDistanceMatching — all default for level 6.
- Compression level: fixed at 6.
- Contexts are per-call (ZSTD_createCCtx / ZSTD_freeCCtx), not cached.

Question

Under the above encoder-side configuration, can we rely on:
1. A v1.5.5 decoder correctly decompressing every frame produced by a v1.5.7 encoder, and
2. A v1.5.7 decoder correctly decompressing every frame produced by a v1.5.5 encoder,

for both dictionary-compressed and non-dictionary calls, provided the same raw-content prefix bytes are supplied on both sides?

Reading #3110, #3711, and #3802, our understanding is "yes" — 1.5.6 and 1.5.7 shipped compressor fixes and decoder-tolerance improvements but no wire-format changes. We would appreciate
a one-line confirmation for the record so we can attach it to our internal rollout decision document.

Thank you.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.