JuliaIO / JuliaIO/TranscodingStreams.jl
Should `write` with an `END_TOKEN` call `finalize` on the stream to prevent memory leaks?
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 90
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
The desire here is to close the TranscodingStream without closing the underlying buffer. This is documented in https://juliaio.github.io/TranscodingStreams.jl/latest/examples/#Explicitly-finish-transcoding-by-writing-TOKEN_END-1 and says that you should write a TOKEN_END token to the stream. However, an issue with that is that it only flushes the stream but it doesn't finalize it which leads to memory leaks in code written like:
using CodecZlib
using TranscodingStreams
function leak()
buf = IOBuffer()
data = rand(10^6)
while true
zWriter = ZlibCompressorStream(buf)
write(zWriter, data)
write(zWriter, TranscodingStreams.TOKEN_END)
flush(zWriter)
end
end
leak()
which will indefinitely leak. Manually calling finalize on the zWriter fixes the issue but it is not clear from the documentation that this is required. There are a few possible solutions:
- Attach a finalizer to the stream that calls
finalize. This is not ideal because you want a more eager cleanup than whenever the GC gets to it. - Make writing a
TOKEN_ENDcallfinalizeon the stream. - Make writing a
TOKEN_ENDset the stream mode to:closedand thereby allowingcloseon the wrapper stream to not close the underlying wrapped stream : https://github.com/JuliaIO/TranscodingStreams.jl/blob/2fac97171c2ff7b6828f49e34c22eb929cb672a1/src/stream.jl#L174-L183
Alternatively, it is also possible that the code that shows the leak above is "faulty" but generally, normal Julia code shouldn't leak like this so at least a finalizer might be a good idea.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the TOKEN_END example in the documentation and the stream lifecycle code in src/stream.jl around lines 174-183. Reproduce the leak using the Julia snippet, then compare the proposed finalize, finalizer, and closed-mode behaviors. Done should preserve the underlying buffer while eagerly releasing the transcoding stream's resources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100