Ligatures are affecting scroll performance

I am using the FiraCode font with macligatures set. When scrolling down, MacVim seems to hiccup a bit, but when I scroll up it is very smooth. When I do not set macligatures it scrolls fine (but doesn't show the cool characters). I am testing the scroll speed using both my touchpad and jk keys and my key repeat is set at 1 which is 15ms.

This was originally posted in FiraCode's issues, but the solution to the problem probably lies in improving macligatures support.

OSX 10.11, MacbookPro Retina

Followed the instructions to

set macligatures
set guifont=Fira\ Code:h12

Doing some dumb testing against Activity Monitor, it looks like the CPU load when I scroll down is much lower (~%60) than when I scroll up (150-200%), so the downscrolling seems to be inhibiting the scroll rather than taking on a huge CPU load and causing framerate hiccups.

Looking around for some possibly related topics:
Guy with the same issue (Scroll to Amadeus Demarzi)
thread related to listchars

Start by reproducing the report in MacVim on macOS with Fira Code, using set macligatures and the listed guifont setting, then compare upward and downward scrolling with and without ligatures. Read the linked FiraCode issue and related listchars thread for context; done means scrolling no longer shows the reported directional slowdown when macligatures is enabled.