G711 aLaw or Opus audio over RTSP streaming with video returns Error

Version

Media3 main branch

More version details

I have a RTSP streaming with a G711 alaw audio and video where I get errors even if using G711 aLaw or Opus; I've seen already another issue that was exposing a similar problem about audio and video over RTSP streaming (https://github.com/androidx/media/issues/893)...
Steps I did :

• First , I builded ffmpeg library on Ubuntu using Android , including ffmpeg native library ( and enabling libopus too because I am trying with standard opus audio too, not the one encapsulated into ogg ) compiled and installed previously in the system , and then generating those .so files. - Then , In my build.gradle I declared as follow:

implementation fileTree(dir: 'libs', include: ['libffmpeg.aar'], exclude: [])

implementation "androidx.media3:media3-exoplayer:1.5.1"
implementation "androidx.media3:media3-common:1.5.1"
implementation "androidx.media3:media3-container:1.5.1"
implementation "androidx.media3:media3-exoplayer-rtsp:1.5.1"
implementation "androidx.media3:media3-exoplayer-dash:1.5.1"
implementation "androidx.media3:media3-exoplayer-hls:1.5.1"
implementation "androidx.media3:media3-ui:1.5.1"
implementation 'androidx.media3:media3-transformer:1.5.1'
implementation "androidx.media3:media3-datasource:1.5.1"
//implementation "androidx.media3:media3-decoder:1.5.1"
implementation "androidx.media3:media3-effect:1.5.1"
implementation "androidx.media3:media3-extractor:1.5.1"

to load the AAR file I picked from output directory out under Ubuntu

• In MainActivity I loaded those .so files :

static {
	try {
		System.loadLibrary("ffmpegjni");
	} catch (java.lang.UnsatisfiedLinkError e ) {
		Log.d("MainActivity", "Couldn't load external libraries");
		e.printStackTrace();
	}
}

which I picked from cxx files out under Ubuntu.

• I implemented loading RTSP as follow :

private boolean LoadVideo(Boolean reload) {
	final boolean reload_done = false;

	WaitDialog = AppUtil.showProgress(getActivity());

	//args = new ArrayList<>();
	//args.add("-vvv");
	//args.add("--network-caching="+mNetworkCaching);
	//args.add("--drop-late-frames");
	//args.add("--no-audio-time-stretch");
	//args.add("--autoscale");
	//args.add("--fullscreen");
	//args.add("--vout=android-display");

	try {
		final String url_nedia = "rtsp://" + mUser + ":" + mPasswd + "@" + mIP + ":8554/" + mHostName;

		if ((mediaPlayer != null) && !mediaPlayer.isReleased() && mediaPlayer.isPlaying()) {
			mediaPlayer.stop();
			mediaPlayer.release();
		}

		mediaItem = new MediaItem.Builder()
			.setUri(Uri.parse(url_nedia.toString()))
			.build();

		defaultDataSource = new DefaultDataSource.Factory(
			this.getContext()
		);

		Uri uri = Uri.parse(url_nedia.toString());
		rtspMediaSource = new RtspMediaSource.Factory()
				.setTimeoutMs(1000)
				.createMediaSource(MediaItem.fromUri(uri));

		defaultRenderersFactory = new DefaultRenderersFactory(this.getContext())
				.setExtensionRendererMode(DefaultRenderersFactory.EXTENSION_RENDERER_MODE_PREFER);

		scaleAndRotateTransformation = new ScaleAndRotateTransformation.Builder().setRotationDegrees(parseFloat(degreesRTSP)).build();

		// .setBufferDurationsMs(32 * 1024, 64 * 1024, 1024, 1024)
		//.setBufferDurationsMs(48 * 1024, 3060 * 1024, 48 * 1024, 48 * 1024)
		customLoadControl = new DefaultLoadControl.Builder()
				.setBufferDurationsMs(32 * 1024, 64 * 1024, 1024, 1024)
				.build();

		//.setRenderersFactory(defaultRenderersFactory)
		//.setLoadControl(customLoadControl)
		mediaPlayer = new ExoPlayer.Builder(this.getContext())
		        .setRenderersFactory(defaultRenderersFactory)
				.build();

				//.setSkipSilenceEnabled(true)
				//.setVideoChangeFrameRateStrategy(C.FIRST_FRAME_NOT_RENDERED)
				//.setPauseAtEndOfMediaItems(true)
				//.setSuppressPlaybackOnUnsuitableOutput(true)
				//.setUseLazyPreparation(true)

		playerView = getView().findViewById(R.id.playerView);
		playerView.setPlayer(mediaPlayer);

		mediaPlayer.addAnalyticsListener(new EventLogger());
		mediaPlayer.setMediaItem(mediaItem);
		//mediaPlayer.setMediaSource(rtspMediaSource);
		mediaPlayer.setVideoScalingMode(C.VIDEO_SCALING_MODE_DEFAULT);
		mediaPlayer.setVideoEffects(ImmutableList.of(scaleAndRotateTransformation));
		mediaPlayer.prepare();
		mediaPlayer.setPlayWhenReady(true);
		mediaPlayer.play();

	} catch(Exception e) {
		throw new RuntimeException("Media3 error: " + e.toString());
	}

	if (WaitDialog != null) {
		WaitDialog.dismiss();
	}
	return reload_done;
}

but when I execute the function I obtain in logcat an error about "Audio sink error" and I see playing video only...

I attach the logcat result for G711 audio, where you can see the external library ffmpeg is correctly loaded ( Loaded FfmpegVideoRenderer , Loaded FfmpegAudioRenderer )

If I set Opus (not the encapsulated inside OGG, but standard Opus) , I get instead something like and index out of bounds and I can't even see playing video but just a black screen. I attach also the logcat using opus audio too. Maybe it's a bug or maybe it should need some extra parameters?
( If I'd use VLC libraries passing those parameters as args commented in the code , then it works well )

LogCat_RENDER_MODE_PREFER_FFMpegReader_G711_alaw.txt
LogCat_RENDER_MODE_PREFER_FFMpegReader_opus.txt

Devices that reproduce the issue

any recent android device

Devices that do not reproduce the issue

None (so far)

Reproducible in the demo app?

Not tested

Reproduction steps

I could reproduce that from a custom device only

Expected result

The video should play with audio (synced to the video)

Actual result

If I play with G711 aLaw I get an Audio sink error and plays video only
If I play with Opus I get an:
Caused by: java.lang.IndexOutOfBoundsException: Index: 0
at java.util.Collections$EmptyList.get(Collections.java:4888)
at androidx.media3.decoder.ffmpeg.FfmpegAudioDecoder.getExtraData(FfmpegAudioDecoder.java:188)

See attached LogCat files

Media

LogCat_RENDER_MODE_PREFER_FFMpegReader_G711_alaw.txt
LogCat_RENDER_MODE_PREFER_FFMpegReader_opus.txt

since it's from a custom external device where from I can't save a RTSP file format , I add logcat files and if needed, I could provide a PCAP of wireshark sniffing test

Bug Report

• You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.