Several drmModeSetCrtc() calls cause video-lockup.

ABOUT THE TEST SYSTEM

-Running standard Linux graphics stack in full KMS mode (BRCM legacy libraries removed):

/boot/config.txt:
dtoverlay=vc4-kms-v3d

-Latest official kernel, 64bit kernel and libs:

/boot/config.txt:
arm_64=1

pi@raspberrypi:~/src/LBE_DOCS/DRM_GBM_VIDEOMODE_CHANGES_TEST $ uname -r
5.4.51-v8+

-Latest stable libDRM and MESA installed:

mesa-20.1.3
libdrm-2.4.102

PROBLEM / BUG

Calling drmModeSetCrtc() several times causes a video-lockup (monitors report "no-signal"), which causes programs that dynamically change physical video mode to fail.
An error regarding the problem cause is shown in dmesg:

[ 2974.688398] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:74:crtc-2] flip_done timed out

TEST PROGRAM

I have made a small program that changes the physical display mode using drmModeSetCrtc() repeatedly, thus causing the video lock-up and the aforementioned dmesg error.
It's available here (please adjust the mode_width values in main.c to something compatible with your monitor):

https://github.com/vanfanel/LBE_DOCS/tree/master/DRM_GBM_VIDEOMODE_CHANGES_TEST

Relevant program sections are:

1. The drmModeSetCrtc() call:
   https://github.com/vanfanel/LBE_DOCS/blob/82ac9fa3dd2a03b9d5444936ae7caa4dd7af2ea2/DRM_GBM_VIDEOMODE_CHANGES_TEST/context.c#L555

2. The pageflip issuing call:
   https://github.com/vanfanel/LBE_DOCS/blob/82ac9fa3dd2a03b9d5444936ae7caa4dd7af2ea2/DRM_GBM_VIDEOMODE_CHANGES_TEST/context.c#L562

3. The pageflip waiting function, called here:
   https://github.com/vanfanel/LBE_DOCS/blob/82ac9fa3dd2a03b9d5444936ae7caa4dd7af2ea2/DRM_GBM_VIDEOMODE_CHANGES_TEST/context.c#L571

As you will see, whenever a drmModeSetCrtc() call is done, we have waited for previously issued pageflip to complete. We wait here, in the GBM+EGL surfaces destruction function:
https://github.com/vanfanel/LBE_DOCS/blob/82ac9fa3dd2a03b9d5444936ae7caa4dd7af2ea2/DRM_GBM_VIDEOMODE_CHANGES_TEST/context.c#L234

So, all in all, it seems to be a problem with drmModeSetCrtc() specific to the VC4 libDRM module, or the kernel. It doesn't happen on intel or amdgpu.

STEPS TO REPRODUCE

Run the test program (https://github.com/vanfanel/LBE_DOCS/tree/master/DRM_GBM_VIDEOMODE_CHANGES_TEST) repeatedly. It will eventually cause the video lock-up.

Start with the linked DRM_GBM_VIDEOMODE_CHANGES_TEST/context.c sections around lines 234, 555, 562, and 571, then reproduce under vc4-kms-v3d using the supplied kernel, Mesa, and libdrm versions. Inspect the drmModeSetCrtc() and pageflip sequence alongside the dmesg flip_done timeout. Done means repeated physical mode changes no longer cause a monitor no-signal lockup or the reported timeout.