Latest commits not reliably detected when branches are fetched

After pushing a commit and getting triggered by a GitLab webhook, GitLab branch source is not reliably detecting that latest commit. We are repeatedly encountering the issue where GitLab branch source reports "no changes detected" for the respective branch in the Jenkins multibranch pipeline event log and consequently does not trigger a build.

Doing some investigation I found the cause of the issue being the GitLab API itself not reliably giving the latest commit for existing branches. The reason behind this is a caching mechanism which was lately introduced into GitLab API when querying multiple branches.