mlt_service_get_frame using movit filters may result in opengl operations w/o valid context

From investigating kdenlive, MLT and movit. Using this ticket for tracking the MLT implementation aspects of the customer goal of GPU accelerated filters. The only GPU accelerated filters (afaik anyways) are those using movit.

• the current MLT contract is: client applications are responsible for establishing the appropriate GL context.
• movit has a similar contract: client applications are responsible for establishing the appropriate GL context.
• MLT can delegate thread creation to client applications for uses of mlt_thread_create but not for consumer_worker_thread.
• the rendering of a movit filter chain cannot be invoked in parallel.
• delegating thread creation to client applications is necessary to establish a valid GL context for the thread. (At least for Qt applications)

All combined, client applications must: manage GL contexts ; use real_time -1, 0, or 1 when using movit for proper behavior.

Considering only movit as currently implemented and OpenGL only: The current implementation works with the restriction on real_time. Considering beyond these to situations like: OpenCL / Vulkan acceleration ; parallel GL rendering. This is not sufficient.

The current MLT and movit contract effectively delegates to the client application management of per-thread acceleration resources. Which I agree with. I think there is the need for two things:

1. MLT delegation of consumer_worker_thread thread creation to client application.

2. Addition of a per-render resource structure. (maybe already exists?) This would not be sharable between threads and is necessary for the client application to provide.

The GL context is a per-thread global which is (thankfully) not a design continued by opencl/vulkan. These have explicit device or command queue values. A per-render resource structure would enable providing client application managed resources to accelerated filters. As well as making explicit the GL context.

Still digging through the current kdenlive integration. Will try to keep these notes updates.

Original ticket:

Digging into the kdenlive issues using GPU effects I determined, experimentally, that MLT was performing OpenGL operations on a thread that was not a render thread. This meant the OpenGL context was either invalid (resulting in a variety of errors) or missing (resulting in a quick exit).

Specifically, this function in kdenlive:

• https://github.com/KDE/kdenlive/blob/refactoring_timeline/src/mltcontroller/clipcontroller.cpp#L662

calls get_frame on an Mlt::Producer. If this producer is a movit/glsl effect then this can result in opengl commands being issued on whatever thread that called get_frame. Which may or may not have a GL context.

When looking into the MLT docs I noticed the warning about multithreaded and opengl. Tho nothing explicit about all application code having to invoke MLT operations from within a rendering thread. Filing here so the caveat is known.

Question tho: should get_frame be thread safe for application code? IMO yes, this is necessary.

Related to this might be relaxing the restrictions on real_time values and the create thread events. Looked fairly straight forward, but not sure if there is some subtlety I'm missing:

• https://github.com/coreyoconnor/mlt/commit/1347f75d5506096dbf176b4f5c6840cf2ea7bbbb

Start with mlt_service_get_frame and the cited kdenlive clipcontroller.cpp call site, then trace consumer_worker_thread, mlt_thread_create, and the movit filter path. Determine how thread creation and per-render resources should be represented for valid OpenGL contexts and future acceleration APIs. Done requires an agreed design and implementation that preserves the stated MLT and movit contracts.