VC4 ISP stuck on upstream

Start by reproducing the failure from the rpi4-isp branch on Raspberry Pi 4B with the stated downstream DTBs, libcamera v0.5.0, and the manual media-ctl format setup. Compare the attached upstream and downstream libcamera and vchiq-mmal logs, then use the bcm2835-isp timeout and vclog assertion as entry points. Done means identifying the missing dependency or upstream-versus-firmware interaction that prevents BUFFER_TO_HOST and documenting a validated fix or next debugging step.

Describe the bug

Hi @6by9 @naushir,

I am trying to upstream support for the VC04 based ISP driver. For this, I have destaged VCHIQ and MMAL drivers, which are already posted upstream.

Rest of the in-progress work is in this branch (based on top of v6.18-rc1):
https://github.com/jailuthra/linux/commits/rpi4-isp

This branch includes vc-sm-cma, misc patches for firmware and char drivers, DMA support for BCM2711, and finally the ISP driver. The vc-sm-cma and bcm2835-isp drivers have been moved outside of staging as part of this branch, for posting upstream.

With this I am able to probe all relevant drivers, and the media graph is populated, but if I try to stream with unicam + ISP using libcamera the stream gets stuck without returning a single frame.

Unicam-only capture runs fine though. From the logs, looks like the ISP does not dequeue any queued buffers. Once the process gets stuck, the firmware is in a hung state and requires a system reboot to recover.

I am using the same devicetree, overlays and firmware as downstream (Bookworm, 6.12). So the only change is on the kernel image and driver modules side.

I have attached the debug logs from libcamera and from vchiq-mmal driver below, for both the working (downstream) and non-working (upstream) case.

My questions are:

1. Am I missing some obvious dependency in my branch, that is required for MMAL and remote ISP driver to function?
2. How can I debug this better? As the kernel side debug logs don't seem to give me details of what's going wrong on the firmware side.

Steps to reproduce the behaviour

1. Clone https://github.com/jailuthra/linux
2. Switch to rpi4-isp branch
3. Compile and install the kernel and modules (using bcm2711_defconfig) without DTBs (use downstream DTBs) to an Raspberry Pi 4B
4. Attach a camera module and run libcamera

Step 4 might not work if you've recently updated libcamera, which returns error on broken media pipelines (with upstream unicam driver).

I am using v0.5.0, where the pipeline handler works fine, but the capture still initially fails due to broken pipe as the format has to be set manually using media-ctl -V on the unicam subdevice. Once that is done, it proceeds to capture raw frames from the sensor and submits them to the ISP.

This whole mechanism works fine on the downstream kernel as well (if forced to use the newer unicam driver), but gets stuck on the ISP dequeue with my upstream branch.

Device (s)

Raspberry Pi 4 Mod. B

System

Which OS and version?

Raspberry Pi reference 2024-07-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 0b115f302a8f1e5bd3523614d7f45b9d447434c7, stage4

Which firmware version?

Aug 20 2025 17:02:58
Copyright (c) 2012 Broadcom
version cd866525580337c0aee4b25880e1f5f9f674fb24 (clean) (release) (start_db)

Logs

libcamera-upstream.txt
libcamera-downstream.txt
vchiq-mmal-downstream.txt
vchiq-mmal-upstream.txt

As you can see the BUFFER_TO_HOST message never arrives in the case of upstream branch, with everything else before that being the exact same.

Once the stream gets stuck, I see prints on dmesg:

[  +0.000003] bcm2835_mmal_vchiq: timed out waiting for sync completion
[  +0.000008] Firmware transaction 0x00030002 timeout
[  +0.000009] bcm2835-isp bcm2835-isp: bcm2835_isp_node_stop_streaming: Failed disabling i/p port, ret -62

And vclog gives me the same timeout, but nothing before that:

darkapex@rpios:~ $ sudo vclog -a
079043.085: assert( ip_isp_timeout fired ) failed; ../../../../../middleware/openmaxil/components/isp.c::ip_isp_timeout line 1894

Additional context

No response