Slower than usual performance using Angrylion

Now, I know Angrylion is a slow plugin by its very nature, and very few people have the computing horsepower needed to run it at full speed on most games, but the libretro version actually performs significantly worse than the zilmar spec version. Even with the VI filters off, I routinely get 10 to 20 less FPS using Angrylion in conjunction with the CXD4 RSP plugin than using the same combination in Project64 2.2, Mupen64 0.5, and even an old version of Mupen64Plus from when it still used the zilmar spec. All three of those emulators perform very similarly, only varying by a VI/s or two.

Now I thought, maybe the zilmar spec versions of both Angrylion and CXD4 have gotten some performance bumps by HatCat somewhere along the way that did not make it into libretro, so I even went through the trouble of finding old versions of those plugins. Indeed, using the first release of HatCat's RSP plugin did result in a drop in performance (older versions of his fork of Angrylion's plugin yielded no appreciable drop), but only by about 6 to 10 VI/s, still nowhere near as slow as their libretro counterparts.

My main benchmark for testing was two screens in Mario 64: the very first screen after you select your file and Mario is standing outside the castle, and the screen as soon as you enter the castle. The latter was most informative, as it's a rare screen where I actually just barely get full speed using the zilmar spec plugins, whereas the libretro version is nearly twice as slow, getting me around 34fps. The outside part is compared here:

Not as much difference here as in the screen inside the castle for whatever reason, but still significant.

Another game I tested was Starcraft 64, where I expected to maybe get full speed with libretro since it's 2D and all. Once again, there was a significant difference: the zilmar spec version gets full speed in-game, whereas libretro only gets around 45fps.

Any thoughts as to what could possibly account for such a difference in performance?

No source file or test is named. Start by reproducing the Mario 64 castle and Starcraft 64 comparisons with the libretro and zilmar-spec Angrylion/CXD4 configurations, then inspect the relevant performance paths and measure the difference. Done means the reported gap is explained and the libretro configuration's performance is verified against those cases.