raspberrypi / raspberrypi/linux

[RPi4] rx_errors increase on default interface

Open
#6,488 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
13.2k
Forks
5.5k
Avg merge
2d 21h
Merged PRs (30d)
21

Description

Describe the bug

I have a cluster a RPi composed of 2 x Pi4 (with official PoE+ hats) + 1 x Pi5 (powered via USB-C)

The default interfaces (BCM2711) of both of the Pi4 of error counter keeps increasing on the default. Linked via the set same of cables and router, the RPi5 has no error after few weeks of runtime, so I believe this is unlikely caused by a faulty equipment.

image

Here is the result of ethtool and uname

<RPi4 #1> # ethtool -S eth0
NIC statistics:
     rx_packets: 7937985429
     tx_packets: 7183614306
     rx_bytes: 5820820488883
     tx_bytes: 4718251881940
     rx_errors: 489032
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 0
     multicast: 0
     rx_64_octets: 130127786
     rx_65_127_oct: 2762379492
     rx_128_255_oct: 927055884
     rx_256_511_oct: 506935297
     rx_512_1023_oct: 156237172
     rx_1024_1518_oct: 3455814450
     rx_vlan_1519_1522_oct: 0
     rx_1522_2047_oct: 0
     rx_2048_4095_oct: 0
     rx_4096_9216_oct: 0
     rx_pkts: 3643582785
     rx_bytes: 3550714151
     rx_multicast: 67124
     rx_broadcast: 20703212
     rx_fcs: 0
     rx_control: 0
     rx_pause: 0
     rx_unknown: 0
     rx_align: 0
     rx_outrange: 0
     rx_code: 0
     rx_carrier: 0
     rx_oversize: 0
     rx_jabber: 0
     rx_mtu_err: 0
     rx_good_pkts: 3643582785
     rx_unicast: 3622812449
     rx_ppp: 0
     rx_crc: 1
     tx_64_octets: 53378213
     tx_65_127_oct: 3053523655
     tx_128_255_oct: 943630826
     tx_256_511_oct: 203920798
     tx_512_1023_oct: 176731901
     tx_1024_1518_oct: 2752428913
     tx_vlan_1519_1522_oct: 0
     tx_1522_2047_oct: 0
     tx_2048_4095_oct: 0
     tx_4096_9216_oct: 0
     tx_pkts: 2888647010
     tx_multicast: 4202
     tx_broadcast: 3297560
     tx_pause: 0
     tx_control: 0
     tx_fcs_err: 0
     tx_oversize: 0
     tx_defer: 0
     tx_excess_defer: 0
     tx_single_col: 0
     tx_multi_col: 0
     tx_late_col: 0
     tx_excess_col: 0
     tx_frags: 0
     tx_total_col: 0
     tx_jabber: 0
     tx_bytes: 1411079996
     tx_good_pkts: 2888647010
     tx_unicast: 2885345248
     rx_runt_pkts: 0
     rx_runt_valid_fcs: 0
     rx_runt_inval_fcs_align: 0
     rx_runt_bytes: 0
     rbuf_ovflow_cnt: 0
     rbuf_err_cnt: 0
     mdf_err_cnt: 7012555
     alloc_rx_buff_failed: 0
     rx_dma_failed: 0
     tx_dma_failed: 0
     tx_realloc_tsb: 0
     tx_realloc_tsb_failed: 0
     txq0_packets: 1309147666
     txq0_bytes: 924442884484
     rxq0_bytes: 0
     rxq0_packets: 0
     rxq0_errors: 0
     rxq0_dropped: 0
     txq1_packets: 1923289003
     txq1_bytes: 1038838328063
     rxq1_bytes: 0
     rxq1_packets: 0
     rxq1_errors: 0
     rxq1_dropped: 0
     txq2_packets: 1490181366
     txq2_bytes: 913337472626
     rxq2_bytes: 0
     rxq2_packets: 0
     rxq2_errors: 0
     rxq2_dropped: 0
     txq3_packets: 1261553296
     txq3_bytes: 878191410838
     rxq3_bytes: 0
     rxq3_packets: 0
     rxq3_errors: 0
     rxq3_dropped: 0
     txq16_packets: 1199442975
     txq16_bytes: 963441785929
     rxq16_bytes: 5820820488883
     rxq16_packets: 7937985429
     rxq16_errors: 489032
     rxq16_dropped: 0
<RPi4 #1> # uname -a
Linux ... 5.15.0-1055-raspi #58-Ubuntu SMP PREEMPT Sat May 4 03:52:40 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
<RPi4 #2> # ethtool -S eth0
NIC statistics:
     rx_packets: 6676483955
     tx_packets: 5314682323
     rx_bytes: 5812562256738
     tx_bytes: 4169991443919
     rx_errors: 769692
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 0
     multicast: 0
     rx_64_octets: 62625474
     rx_65_127_oct: 2083186789
     rx_128_255_oct: 317813688
     rx_256_511_oct: 509009813
     rx_512_1023_oct: 200819326
     rx_1024_1518_oct: 3503727525
     rx_vlan_1519_1522_oct: 0
     rx_1522_2047_oct: 0
     rx_2048_4095_oct: 0
     rx_4096_9216_oct: 0
     rx_pkts: 2382215319
     rx_bytes: 3254794218
     rx_multicast: 22835
     rx_broadcast: 7360989
     rx_fcs: 0
     rx_control: 0
     rx_pause: 0
     rx_unknown: 0
     rx_align: 0
     rx_outrange: 0
     rx_code: 0
     rx_carrier: 0
     rx_oversize: 0
     rx_jabber: 0
     rx_mtu_err: 0
     rx_good_pkts: 2382215319
     rx_unicast: 2374831495
     rx_ppp: 0
     rx_crc: 3
     tx_64_octets: 50562220
     tx_65_127_oct: 1963043132
     tx_128_255_oct: 298283415
     tx_256_511_oct: 419801169
     tx_512_1023_oct: 89159539
     tx_1024_1518_oct: 2493832848
     tx_vlan_1519_1522_oct: 0
     tx_1522_2047_oct: 0
     tx_2048_4095_oct: 0
     tx_4096_9216_oct: 0
     tx_pkts: 1019715027
     tx_multicast: 1351
     tx_broadcast: 34
     tx_pause: 0
     tx_control: 0
     tx_fcs_err: 0
     tx_oversize: 0
     tx_defer: 0
     tx_excess_defer: 0
     tx_single_col: 0
     tx_multi_col: 0
     tx_late_col: 0
     tx_excess_col: 0
     tx_frags: 0
     tx_total_col: 0
     tx_jabber: 0
     tx_bytes: 3960365824
     tx_good_pkts: 1019715027
     tx_unicast: 1019713642
     rx_runt_pkts: 0
     rx_runt_valid_fcs: 0
     rx_runt_inval_fcs_align: 0
     rx_runt_bytes: 0
     rbuf_ovflow_cnt: 0
     rbuf_err_cnt: 0
     mdf_err_cnt: 2479011
     alloc_rx_buff_failed: 0
     rx_dma_failed: 0
     tx_dma_failed: 0
     tx_realloc_tsb: 0
     tx_realloc_tsb_failed: 0
     txq0_packets: 842743825
     txq0_bytes: 619556325880
     rxq0_bytes: 0
     rxq0_packets: 0
     rxq0_errors: 0
     rxq0_dropped: 0
     txq1_packets: 1091289391
     txq1_bytes: 863464869700
     rxq1_bytes: 0
     rxq1_packets: 0
     rxq1_errors: 0
     rxq1_dropped: 0
     txq2_packets: 1021091507
     txq2_bytes: 847011567597
     rxq2_bytes: 0
     rxq2_packets: 0
     rxq2_errors: 0
     rxq2_dropped: 0
     txq3_packets: 1047747286
     txq3_bytes: 767982480777
     rxq3_bytes: 0
     rxq3_packets: 0
     rxq3_errors: 0
     rxq3_dropped: 0
     txq16_packets: 1311810314
     txq16_bytes: 1071976199965
     rxq16_bytes: 5812562256738
     rxq16_packets: 6676483955
     rxq16_errors: 769692
     rxq16_dropped: 
<RPi4 #2> # uname -a
Linux ... 5.15.0-1063-raspi #66-Ubuntu SMP PREEMPT Fri Aug 30 16:09:00 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
<RPi5> # ethtool -S eth0
NIC statistics:
     tx_octets: 5366741369327
     tx_frames: 5981882664
     tx_broadcast_frames: 1158323
     tx_multicast_frames: 2255
     tx_pause_frames: 0
     tx_64_byte_frames: 5028095
     tx_65_127_byte_frames: 1641453597
     tx_128_255_byte_frames: 348987236
     tx_256_511_byte_frames: 592524279
     tx_512_1023_byte_frames: 198685983
     tx_1024_1518_byte_frames: 3195209765
     tx_greater_than_1518_byte_frames: 0
     tx_underrun: 0
     tx_single_collision_frames: 0
     tx_multiple_collision_frames: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     tx_deferred_frames: 0
     tx_carrier_sense_errors: 0
     rx_octets: 4289326609033
     rx_frames: 4754880551
     rx_broadcast_frames: 4203894
     rx_multicast_frames: 1793053
     rx_pause_frames: 0
     rx_64_byte_frames: 9950310
     rx_65_127_byte_frames: 1389109633
     rx_128_255_byte_frames: 271809139
     rx_256_511_byte_frames: 376097102
     rx_512_1023_byte_frames: 110079805
     rx_1024_1518_byte_frames: 2597836732
     rx_greater_than_1518_byte_frames: 0
     rx_undersized_frames: 0
     rx_oversize_frames: 0
     rx_jabbers: 0
     rx_frame_check_sequence_errors: 0
     rx_length_field_frame_errors: 0
     rx_symbol_errors: 0
     rx_alignment_errors: 0
     rx_resource_errors: 0
     rx_overruns: 0
     rx_ip_header_checksum_errors: 0
     rx_tcp_checksum_errors: 0
     rx_udp_checksum_errors: 0
     q0_rx_packets: 4754956911
     q0_rx_bytes: 4203796003496
     q0_rx_dropped: 0
     q0_tx_packets: 2668351751
     q0_tx_bytes: 5124159748186
     q0_tx_dropped: 0
<RPi5> # uname -a
Linux ... 6.6.51+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/Linux
Steps to reproduce the behaviour

There is no clear step to reproduce, nothing in neither journalctl nor dmesg. The value gradually grows without any viability of where the problem is at.

Device (s)

Raspberry Pi 4 Mod. B

System

(For RPi4 #2)

# vcgencmd version
Oct 26 2022 11:09:05 
Copyright (c) 2012 Broadcom
version c72ad6b26ff40c91ef776b847436094ee63fabee (clean) (release) (start)
# uname -a
Linux ... 5.15.0-1063-raspi #66-Ubuntu SMP PREEMPT Fri Aug 30 16:09:00 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
Logs

No response

Additional context

Appreciate it isn't much to go about. Please let me know if there is more data I could provide to help finding this bug. Note that all RPi run Calico as overlay network, in case this may add clue of what would be going wrong?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by reproducing the comparison on Raspberry Pi 4 and Pi 5, recording ethtool -S eth0, uname -a, vcgencmd version, dmesg, and journalctl output over time; use the Ethernet statistics and kernel versions to narrow the affected driver path. Done means identifying a reproducible cause for the growing rx_errors or documenting the missing evidence needed to continue.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.