dotnet / dotnet/msbuild

Binlog reading - grace checking of GZip signature

Open
#9,567 1 comment 0 reactions 0 assignees View on GitHub
help wanted Priority:3 triaged
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 8h
Merged PRs (30d)
141

Description

### Context

Grace reporting if non-Gzip stream encountered

### Background

Idea by @KirillOsenkov:

> One other thought I had that's not necessarily for this PR, but somewhat related. Right now all binlog files start with `1F 8B 08` which is the GZip stream signature bytes (08 indicating Deflate compression). If we change the compression method for the outer stream at some point in the future, the envelope format will likely have a different signature. Long term I'd love to investigate Zstd or Lzma compression, I remember I saw more than 2x improvement in binlog sizes when I uncompressed Gzip/Deflate and recompressed with 7-Zip.
>
> Wondering if we could prepare for that future today by gracefully showing a message if the binlog file doesn't start with `1F 8B 08`.
>
> Currently we handle it relatively well: `There was an exception while reading the log file: Found invalid data while decoding.` Wondering if there's anything else we can do here, or probably not?

### Further Thoughts

This might not be that helpful now as we support only a GZip streams. But would we (or contributor :-)) decide to add support for other compression mechanisms - it would become a must. Then we'd probably use the signature to flip between the deflation streams implementations (or throw more descriptively if unknown).

Possible support of uncompressed stream might be nice as well - I found myself quite a few times uncompressing the binlog to troubleshoot various issues with it, but then being unable to use it as is.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.