JuliaIO / JuliaIO/TranscodingStreams.jl

Get length of compressed stream so far without closing stream?

Open
#92 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
90
Forks
25
PR merge metrics
No merged PRs in 30d

Description

I would like to get the length of the compressed stream up to now but without closing the stream or affecting continued compression. I understand most Codecs might not support this, given their internal block lengths etc, but maybe there are ways to get related/close to this behavior?

The use case is something like this:

  • We have a very long string/stream which has been compressed already, C(s_long)
  • We now have a set of N shorter strings S_shorts = [s1, ..., sN] and we want to calculate map(length, [C(s_long * s1), ..., C(s_long * sN)]) but without having to redo the whole C(s_long) compression for each of the shorter strings si (since calculating C(s_long) might be costly in time).
  • Note that we only need the lengths of all the C(s_long * si), not their actual bytes.

Any ideas how this can be done as fast as possible? :)

Currently I basically do a Huffman coding/tree or dictionary-based compression by hand and can thus save the intermediate tree/dictionary between each consideration of the short strings, but it would be nice if there is a way to use more advanced compressors like the CodecX ones in the TranscodingStreams framework.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the TranscodingStreams codec interfaces and the CodecX implementations mentioned in the issue. Check whether a supported operation can report compressed length for a prefix without closing or changing continued compression; done means documenting a feasible interface or confirming codec limitations and an example.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.