writeq() on 64bit 5.10.XX branch does not issue PCIe cycle, switching to two writel() does work.

Start with megasas_write_64bit_req_desc and compare the writeq() path with the two writel() path shown in the report. Reproduce with compile.sh and sudo insmod megaraid_sas.ko on the Compute Module 4 setup, then compare the failure and working logs. Done means the driver initializes successfully with writeq() on the affected 64-bit Raspberry Pi kernel without a platform-specific workaround.

Describe the bug
The megaraid_sas driver utilizes the writeq() to perform the PCIe cycle to its adapter. On the 64bit Pi OS kernel 5.10.11 , when loading the driver, the load hangs. The hang was debugged down to a cycle sent to the adapter, but the adapter never receiving the PCIe command. Switching from writeq() to 2 writel() allows the PCIe cycle to be received by the adapter. This driver has been tested to work with many other ARM 64 bit configurations and it would not make sense to impose the performance penalty into the driver for only the RPi.

To reproduce
List the steps required to reproduce the issue.
./compile.sh
sudo insmod megaraid_sas.ko
driver hangs.

if we apply a patch to the driver to change from writeq to two writel., same steps above work.

Driver code in question:
/**

• megasas_write_64bit_req_desc - PCI writes 64bit request descriptor
• @instance: Adapter soft state
• @req_desc: 64bit Request descriptor
  */
  static void
  megasas_write_64bit_req_desc(struct megasas_instance *instance,
  union MEGASAS_REQUEST_DESCRIPTOR_UNION *req_desc)
  {
  #if defined(writeq) && defined(CONFIG_64BIT)
  u64 req_data = (((u64)le32_to_cpu(req_desc->u.high) << 32) |
  le32_to_cpu(req_desc->u.low));
  writeq(req_data, &instance->reg_set->inbound_low_queue_port);
  #else
  unsigned long flags;
  spin_lock_irqsave(&instance->hba_lock, flags);
  writel(le32_to_cpu(req_desc->u.low),
  &instance->reg_set->inbound_low_queue_port);
  writel(le32_to_cpu(req_desc->u.high),
  &instance->reg_set->inbound_high_queue_port);
  #if !((defined(RHEL_MAJOR) && (RHEL_MAJOR == 8) && (RHEL_MINOR >= 2)) ||
  (LINUX_VERSION_CODE >= KERNEL_VERSION(5,2,0)))
  mmiowb();
  #endif
  spin_unlock_irqrestore(&instance->hba_lock, flags);
  #endif

Expected behaviour
Broadcom Adapter is expecting a kernel modification to correctly handle the writeq(), so that it does not need specific platform workarounds in their driver.

Actual behaviour
writeq() is not performing the 64 bit cycle on PCIe.

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW

Compute Module 4 Lite with 4 GB RAM, WiFi/Bluetooth

• Which OS and version (cat /etc/rpi-issue)?

Raspberry Pi reference 2020-08-20
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 7252c154838ec5b4576f29c996ac8fe3750cae12, stage2

5.10.11 and 5.10.13 , with KConfig change to allow CONFIG_IRQ_POLL=y . necessary to compile the megaraid_sas driver for ARM.

• Which firmware version (vcgencmd version)?
  Aug 19 2020 17:38:16
  Copyright (c) 2012 Broadcom
  version e90cba19a98a0d1f2ef086b9cafcbca00778f094 (clean) (release) (start)

• Which kernel version (uname -a)?

Linux raspberrypi 5.10.14-v8+ #3 SMP PREEMPT Fri Feb 12 16:47:24 GMT 2021 aarch64 GNU/Linux

Logs

Failure logs (uses writeq() ):
[ 217.903395] megaraid_sas: loading out-of-tree module taints kernel.
[ 217.909373] megasas: 07.718.00.00
[ 217.909863] pci 0000:00:00.0: enabling device (0000 -> 0002)
[ 217.909899] megaraid_sas 0000:01:00.0: enabling device (0000 -> 0002)
[ 217.910589] megaraid_sas 0000:01:00.0: BAR:0x0 BAR's base_addr(phys):0x0x0000000600200000 mapped virt_addr:0x00000000a7882b4e
[ 217.910605] megaraid_sas 0000:01:00.0: Waiting for FW to come to ready state
[ 218.047987] megaraid_sas 0000:01:00.0: FW now in Ready state
[ 218.047999] megaraid_sas 0000:01:00.0: 63 bit DMA mask and 63 bit consistent mask
[ 218.049878] megaraid_sas 0000:01:00.0: firmware supports msix : (128)
[ 218.050222] megaraid_sas 0000:01:00.0: requested/available msix 5/5
[ 218.050231] megaraid_sas 0000:01:00.0: current msix/online cpus : (5/4)
[ 218.050239] megaraid_sas 0000:01:00.0: RDPQ mode : (enabled)
[ 218.050248] megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 3053 LDIO threshold: 0
[ 218.263380] megaraid_sas 0000:01:00.0: Performance mode :Latency (latency index = 1)
[ 218.263392] megaraid_sas 0000:01:00.0: FW supports sync cache : Yes
[ 218.263405] megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[ 218.263412] megaraid_sas 0000:01:00.0: BRCM DBG: IOC INIT frame addr: 1463f8000 MFI command addr: 14fdf4000

[ 470.267887] megaraid_sas 0000:01:00.0: Init cmd return status FAILED for SCSI host 0
[ 470.453057] megaraid_sas 0000:01:00.0: Failed from megasas_init_fw 6773

working logs (using writel() )
[ 138.129230] megaraid_sas: loading out-of-tree module taints kernel.
[ 138.135261] megasas: 07.718.00.00
[ 138.135732] pci 0000:00:00.0: enabling device (0000 -> 0002)
[ 138.135767] megaraid_sas 0000:01:00.0: enabling device (0000 -> 0002)
[ 138.136656] megaraid_sas 0000:01:00.0: BAR:0x0 BAR's base_addr(phys):0x0x0000000600200000 mapped virt_addr:0x00000000238916d8
[ 138.136677] megaraid_sas 0000:01:00.0: Waiting for FW to come to ready state
[ 139.400492] megaraid_sas 0000:01:00.0: FW now in Ready state
[ 139.400515] megaraid_sas 0000:01:00.0: 63 bit DMA mask and 63 bit consistent mask
[ 139.402988] megaraid_sas 0000:01:00.0: firmware supports msix : (128)
[ 139.403516] megaraid_sas 0000:01:00.0: requested/available msix 5/5
[ 139.403531] megaraid_sas 0000:01:00.0: current msix/online cpus : (5/4)
[ 139.403543] megaraid_sas 0000:01:00.0: RDPQ mode : (enabled)
[ 139.403559] megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 3053 LDIO threshold: 0
[ 139.605418] megaraid_sas 0000:01:00.0: Performance mode :Latency (latency index = 1)
[ 139.605440] megaraid_sas 0000:01:00.0: FW supports sync cache : Yes
[ 139.605452] megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[ 139.716509] megaraid_sas 0000:01:00.0: FW provided supportMaxExtLDs: 1 max_lds: 64
[ 139.716518] megaraid_sas 0000:01:00.0: controller type : MR(4096MB)
[ 139.716523] megaraid_sas 0000:01:00.0: Online Controller Reset(OCR) : Enabled
[ 139.716530] megaraid_sas 0000:01:00.0: Secure JBOD support : Yes
[ 139.716530] megaraid_sas 0000:01:00.0: NVMe passthru support : Yes
[ 139.716539] megaraid_sas 0000:01:00.0: FW provided TM TaskAbort/Reset timeout : 6 secs/60 secs
[ 139.716543] megaraid_sas 0000:01:00.0: PCI Lane Margining support : No
[ 139.716550] megaraid_sas 0000:01:00.0: JBOD sequence map support : Yes
[ 139.745384] megaraid_sas 0000:01:00.0: NVME page size : (4096)
[ 139.745532] megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[ 139.745532] megaraid_sas 0000:01:00.0: INIT adapter done
[ 139.747717] megaraid_sas 0000:01:00.0: Snap dump wait time : 15
[ 139.747735] megaraid_sas 0000:01:00.0: pci id : (0x1000)/(0x0014)/(0x1000)/(0x9460)
[ 139.747740] megaraid_sas 0000:01:00.0: unevenspan support : no
[ 139.747740] megaraid_sas 0000:01:00.0: firmware crash dump : no
[ 139.747740] megaraid_sas 0000:01:00.0: JBOD sequence map : enabled
[ 139.747898] megaraid_sas 0000:01:00.0: Max firmware commands: 3052 shared with nr_hw_queues = 4
[ 139.747904] scsi host0: Broadcom SAS based MegaRAID driver
[ 139.769816] scsi 0:0:97:0: Enclosure LSI VirtualSES 03 PQ: 0 ANSI: 7
[ 139.778076] scsi 0:0:97:0: Attached scsi generic sg0 type 13
[ 139.789480] scsi 0:2:0:0: Direct-Access AVAGO MR9460-16i 5.16 PQ: 0 ANSI: 5
[ 139.790170] sd 0:2:0:0: [sda] 700121088 512-byte logical blocks: (358 GB/334 GiB)
[ 139.790216] sd 0:2:0:0: [sda] Write Protect is off
[ 139.790223] sd 0:2:0:0: [sda] Mode Sense: 1f 00 00 08
[ 139.790295] sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 139.790410] sd 0:2:0:0: Attached scsi generic sg1 type 0
[ 139.790606] sd 0:2:0:0: [sda] Optimal transfer size 196608 bytes
[ 139.791735] scsi 0:2:1:0: Direct-Access AVAGO MR9460-16i 5.16 PQ: 0 ANSI: 5
[ 139.792089] sd 0:2:0:0: [sda] Attached SCSI disk
[ 139.792341] sd 0:2:1:0: Attached scsi generic sg2 type 0
[ 139.792558] sd 0:2:1:0: [sdb] 1754529792 512-byte logical blocks: (898 GB/837 GiB)
[ 139.793181] sd 0:2:1:0: [sdb] Write Protect is off
[ 139.793196] sd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08
[ 139.793281] sd 0:2:1:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 139.793594] sd 0:2:1:0: [sdb] Optimal transfer size 786432 bytes
[ 139.799745] sd 0:2:1:0: [sdb] Attached SCSI disk

Additional context
Add any other relevant context for the problem.

This data came from efforts from work with Jeff Geerling:
https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/72