systemd / systemd/systemd

zstd dictionary compression for journald

Open
#12,579 4 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

journal RFE 🎁
Dominant language
C
Stars
16.7k
Forks
4.7k
Avg merge
1d 6h
Merged PRs (30d)
140

Description

Is your feature request related to a problem? Please describe.
Many journal messages are very small, and messages from the same service are likely to contain much redundancy. The compression of small messages can probably be greatly improved by Zstd's dictionaries [1]
My PR #8207 is insufficient (but a good starting point)

Describe the solution you'd like
For each _CMDLINE, journald will build a zstd dictionary. Exactly how journald will do this, and how it will effect the file format is to be determined, however each journal file must be self-contained, and contain all applicable dictionaries.

@alexgartrell Based on PR #8313 Facebook would probably be interested in this. A bounty[2] would motivate me to work on this, and Facebook has put bounties on bugs there before.

[1] https://facebook.github.io/zstd/#small-data
[2] https://www.bountysource.com/ click "Post Bounty"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing PR #8207, the related PR #8313, and the linked Zstandard dictionary documentation. The issue leaves the journald implementation and file-format changes open; done would require per-_CMDLINE dictionaries while keeping each journal file self-contained, with the format and approach agreed first.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
operating-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.