zstd dictionary compression for journald
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 16.7k
- Forks
- 4.7k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 140
Description
Is your feature request related to a problem? Please describe.
Many journal messages are very small, and messages from the same service are likely to contain much redundancy. The compression of small messages can probably be greatly improved by Zstd's dictionaries [1]
My PR #8207 is insufficient (but a good starting point)
Describe the solution you'd like
For each _CMDLINE, journald will build a zstd dictionary. Exactly how journald will do this, and how it will effect the file format is to be determined, however each journal file must be self-contained, and contain all applicable dictionaries.
@alexgartrell Based on PR #8313 Facebook would probably be interested in this. A bounty[2] would motivate me to work on this, and Facebook has put bounties on bugs there before.
[1] https://facebook.github.io/zstd/#small-data
[2] https://www.bountysource.com/ click "Post Bounty"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing PR #8207, the related PR #8313, and the linked Zstandard dictionary documentation. The issue leaves the journald implementation and file-format changes open; done would require per-_CMDLINE dictionaries while keeping each journal file self-contained, with the format and approach agreed first.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, linux
- Domain
- operating-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100