JuliaIO / JuliaIO/TranscodingStreams.jl

Should `write` with an `END_TOKEN` call `finalize` on the stream to prevent memory leaks?

Open
#117 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
90
Forks
25
PR merge metrics
No merged PRs in 30d

Description

The desire here is to close the TranscodingStream without closing the underlying buffer. This is documented in https://juliaio.github.io/TranscodingStreams.jl/latest/examples/#Explicitly-finish-transcoding-by-writing-TOKEN_END-1 and says that you should write a TOKEN_END token to the stream. However, an issue with that is that it only flushes the stream but it doesn't finalize it which leads to memory leaks in code written like:


using CodecZlib
using TranscodingStreams

function leak()
    buf = IOBuffer()
    data = rand(10^6)
    while true
        zWriter = ZlibCompressorStream(buf)
        write(zWriter, data)
        write(zWriter, TranscodingStreams.TOKEN_END)
        flush(zWriter)
    end
end

leak()

which will indefinitely leak. Manually calling finalize on the zWriter fixes the issue but it is not clear from the documentation that this is required. There are a few possible solutions:

Alternatively, it is also possible that the code that shows the leak above is "faulty" but generally, normal Julia code shouldn't leak like this so at least a finalizer might be a good idea.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TOKEN_END example in the documentation and the stream lifecycle code in src/stream.jl around lines 174-183. Reproduce the leak using the Julia snippet, then compare the proposed finalize, finalizer, and closed-mode behaviors. Done should preserve the underlying buffer while eagerly releasing the transcoding stream's resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.