Noises/Crackling when switching to new MIDI track

I set up MIDI so that Wine can make MIDI work for games.
Each time a track needs to be stopped and switch to a new track (changing levels inside the game, or pressing Next on a MIDI player) there are these noises at the start of the track, like if the buffer is still filling (assumption) Sometimes, the noises last for longer (it seems to get worse if you switch MIDI tracks quicker)

Is this a Buffer Size issue? CPU usage? Pulseaudio buffer different than Timidity buffer?

This issue plagues me from old distributions and it's still happening on new hardware. I must be doing something wrong or missing some essential configuration.
Any tips?
Is there a way for perfectly-matching or synchronizing the Timidity buffer size with the buffer size required for the ALSA/Pulseaudio soundcard to work? Is a mismatch between these two the main suspect?

I even recorded footage on OBS to see if the noises would be recorded by the footage, it's here:
https://www.youtube.com/watch?v=t7Yw9a1OWJQ