JuliaIO / JuliaIO/TranscodingStreams.jl
Get length of compressed stream so far without closing stream?
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 90
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
I would like to get the length of the compressed stream up to now but without closing the stream or affecting continued compression. I understand most Codecs might not support this, given their internal block lengths etc, but maybe there are ways to get related/close to this behavior?
The use case is something like this:
- We have a very long string/stream which has been compressed already, C(s_long)
- We now have a set of N shorter strings S_shorts = [s1, ..., sN] and we want to calculate map(length, [C(s_long * s1), ..., C(s_long * sN)]) but without having to redo the whole C(s_long) compression for each of the shorter strings si (since calculating C(s_long) might be costly in time).
- Note that we only need the lengths of all the C(s_long * si), not their actual bytes.
Any ideas how this can be done as fast as possible? :)
Currently I basically do a Huffman coding/tree or dictionary-based compression by hand and can thus save the intermediate tree/dictionary between each consideration of the short strings, but it would be nice if there is a way to use more advanced compressors like the CodecX ones in the TranscodingStreams framework.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the TranscodingStreams codec interfaces and the CodecX implementations mentioned in the issue. Check whether a supported operation can report compressed length for a prefix without closing or changing continued compression; done means documenting a feasible interface or confirming codec limitations and an example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100