Is it possible to enable the videotrack without a pause in the playback?

I'm working on an audio streaming app that uses ExoPlayer and Media3, with all content delivered via HLS streams.

We're now adding video podcasts to the app. I’ve implemented dynamic video track selection so users can enable or disable video based on preference or device state (e.g., screen off), minimizing bandwidth usage when video isn't needed.

I have done it like this:

val builder = exoPlayer.trackSelectionParameters.buildUpon()
if (isAppInForeground && isVideoWantedByUser) {
            builder.clearOverridesOfType(TRACK_TYPE_VIDEO)
        } else {
            builder
                .addOverride(
                    TrackSelectionOverride(
                        videoGroup.mediaTrackGroup, listOf()
                    )
                )
        }

The issue I'm facing is that enabling the video track causes a brief pause in playback—likely due to the video buffering. I'm wondering if there's a way to prioritize audio playback over video, so that enabling video doesn't interrupt the ongoing audio stream. Ideally, video would load in the background and start displaying once buffered, without causing any playback disruption.

Note that my iOS counterpart did a similar implementation and has this working without the disruption.