`./livepeer -orchestrator -transcoder` process dies, reporting `Killed` (possible memory leak)

Bug Description

When operating livepeer in -orchestrator -transcoder mode, with a B sending it a single stream, the process consumes all RAM on the host, reporting Killed as the process fails.

Observed Behaviour

Steps to reproduce the behavior:

1. Download version 0.5.27 of Livepeer wget https://github.com/livepeer/go-livepeer/releases/download/v0.5.27/livepeer-linux-amd64.tar.gz
2. Unzip tar -xzf livepeer-linux-amd64.tar.gz
3. start O/T ./livepeer -orchestrator -transcoder -serviceAddr 127.0.0.1:8935 -v 99
4. start B ./livepeer -broadcaster -httpAddr :8936 -cliAddr :7936 -rtmpAddr :1936 -orchAddr 127.0.0.1:8935 -transcodingOptions P720p30fps16x9,P576p30fps16x9,P360p30fps16x9,P240p30fps16x9,P144p30fps16x9 -v 99
5. start ffmpeg to publish into the B ffmpeg -re -f lavfi -i testsrc=size=1920x1080:rate=30,format=yuv420p -f lavfi -i sine -c:v libx264 -b:v 10000k -x264-params keyint=60 -c:a aac -f flv rtmp://127.0.0.1:1936/test_source
6. run the following to confirm that transcoding is happening: ffplay http://localhost:8936/stream/test_source/P144p30fps16x9.m3u8
7. observe multiple renditions being served by the B
8. Observe the Memory consumption of the O process start increasing at a rate of ~300Mb every 2 seconds.
9. Observe the O process terminating ungracefully, reporting "Killed".

Expected Behavior

I expect the O process to release Memory once it has transcoded and served content back to the B.

Screenshots

Screenshot from System Monitor application on Ubuntu:

Environment

• Livepeer Version: 0.5.27

Additional Comments

Splitting the O/T into O and T

When running with a split O and T setup, the issue also manifests, with the T process being the one Killed.

B: ./livepeer -broadcaster -httpAddr :8936 -cliAddr :7936 -rtmpAddr :1936 -orchAddr 127.0.0.1:8935 -transcodingOptions P720p30fps16x9,P576p30fps16x9,P360p30fps16x9,P240p30fps16x9,P144p30fps16x9
O: ./livepeer -orchestrator -orchSecret hello -serviceAddr 127.0.0.1:8935
T: ./livepeer -transcoder -orchAddr 127.0.0.1:8935 -orchSecret hello -cliAddr :7937
P: ffmpeg -re -f lavfi -i testsrc=size=1920x1080:rate=30,format=yuv420p -f lavfi -i sine -c:v libx264 -b:v 10000k -x264-params keyint=60 -c:a aac -f flv rtmp://127.0.0.1:1936/test_source

Previous versions

Reproduced on the following versions of Livepeer:

0.5.26
0.5.25
0.5.24
0.5.23
0.5.22
[stopped testing]

System Monitor charts running 0.5.22:

Reproduce the growth using the documented -orchestrator -transcoder setup, the split O/T setup, and the ffmpeg test stream, while monitoring the process memory. Trace the O/T transcoding and serving path to identify what remains retained after content is delivered. Done means the process no longer grows by hundreds of megabytes every few seconds or terminates with "Killed".