Video stream always buffering if audio is not playing

I am using exoplayer to play the stream in a miracast sink application, in order to display a device mirroring it's screen. Using a windows PC, I try to cast to my app, but when I try to play the stream, it will get stuck buffering forever, until I play any kind of audio source on the windows PC. Stream works while the sound is playing, but it goes back to buffering the moment I pause all sounds. While doing this test, whenever I pause and play back a youtube video for example, I always get an error in exoplayer (Audio sink error, Unexpected audio track timestamp discontinuity). If I leave the audio source paused for too long, it will just crash due to the buffering.

I have tried using FLAG_IGNORE_AAC_STREAM on the setTsExtractorFlags, and if I do that the stream works completely as expected from windows, but then I cannot ever play any sound.

I need to be able to keep playing video always, ignoring any lack of AAC stream while there is none, but playing that when it does show up in the same stream. Is there any flag or option where I can set something like this up?