oxidecomputer / oxidecomputer/dendrite

Egress (port) counters return 0 even when counters are updated in Tofino Simulator

Open
#35 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

tofino-simulator
Dominant language
Rust
Stars
20
Forks
3
Avg merge
3d 11h
Merged PRs (30d)
4

Description

While all counters are correctly registered in https://github.com/oxidecomputer/dendrite/pull/14, we're not seeing counts increment via the counters API call (in packet tests) for Egress counters:

async fn check_counter_incremented(
    switch: &Switch,
    counter_name: &str,
    baseline: u64,
    expected_increment: u64,
    client_name: Option<&str>,
) -> anyhow::Result<u64> {
    let mut new_value = 0;

    // Poll for the counter value (with timeout)
    for _i in 0..20 {
        std::thread::sleep(std::time::Duration::from_millis(100));
        new_value =
            switch.get_counter(counter_name, client_name).await.unwrap();

        if new_value == baseline + expected_increment {
            return Ok(new_value);
        }
    }

    // Counter didn't increment as expected
    Err(anyhow!(
        "Counter '{}' expected to increase by {} (from {} to {}), but only reached {}",
        counter_name,
        expected_increment,
        baseline,
        baseline + expected_increment,
        new_value
    ))
}
check_counter_incremented(
    switch,
    &port_label_egress1,
    ctr_baseline_egress1,
    1,
    Some("egress"),
)
.await
.unwrap();

However, the same test function works correctly for Ingress port counters. Of note, the simulator logs show that the counter was, indeed, updated:

:04-22 13:08:57.320787: :-:0x3100000000000001:<0,1,3>:Egress : Gateway did provide payload.
:04-22 13:08:57.320797: :-:0x3100000000000001:<0,1,3>:Execute Action: sidecar1745
:04-22 13:08:57.320825: :-:0x3100000000000001:<0,1,3>:Action Results:
:04-22 13:08:57.320834: :-:0x3100000000000001:<0,1,3>: ----- CountFromHashPrimitive -----
:04-22 13:08:57.320842: :-:0x3100000000000001:<0,1,3>: ----- Update counter: Egress.egress_ctr
:04-22 13:08:57.320851: :-:0x3100000000000001:<0,1,3>: VPN : 0 Ram Line : 0 Subword(Including LSBs. Need to right shift based on stats table format) : 0
:04-22 13:08:57.320860: :-:0x3100000000000001:<0,1,3>:Next Table = --END_OF_PIPELINE--
:04-22 13:08:57.320872: :-:0x3100000000000001:<0,1,->:------------ Stage 4 ------------
:04-22 13:08:57.321005: :-:0x3100000000000001:<0,1,4>:Egress : Table tbl_sidecar1742 is not active(inhibited/skipped)

Here's the P4 Egress control. We had not done any work here before (just an empty apply):

control Egress(
	inout sidecar_headers_t hdr,
	inout sidecar_egress_meta_t meta,
	in egress_intrinsic_metadata_t eg_intr_md,
	in egress_intrinsic_metadata_from_parser_t eg_prsr_md,
	inout egress_intrinsic_metadata_for_deparser_t eg_dprsr_md,
	inout egress_intrinsic_metadata_for_output_port_t eg_oport_md
) {

	Counter<bit<64>, PortId_t>(512, CounterType_t.PACKETS_AND_BYTES) egress_ctr;
	Counter<bit<64>, PortId_t>(512, CounterType_t.PACKETS_AND_BYTES) mcast_ctr;
	Counter<bit<32>, PortId_t>(512, CounterType_t.PACKETS) drop_port_ctr;
	Counter<bit<32>, bit<8>>(DROP_REASON_MAX, CounterType_t.PACKETS) drop_reason_ctr;

	apply {
		// Check multicast egress packets by enforcing replication_id usage
		if (eg_intr_md.egress_rid > 0 ||
			(hdr.ipv6.isValid() && (bit<16>)hdr.ipv6.dst_addr[127:112] != 16w0xff02)) {
			mcast_ctr.count(eg_intr_md.egress_port);
        } else if (eg_intr_md.egress_rid == 0 && eg_intr_md.egress_rid_first == 1) {
			// Drop CPU copies (RID=0) to prevent unwanted packets on port 0
			eg_dprsr_md.drop_ctl = 1;
			meta.drop_reason = DROP_MULTICAST_CPU_COPY;
		}

		if (meta.drop_reason != 0) {
			// Handle dropped packets
			drop_port_ctr.count(eg_intr_md.egress_port);
			drop_reason_ctr.count(meta.drop_reason);
		} else {
			egress_ctr.count(eg_intr_md.egress_port);
		}
	}
}

Checklist

  • Test if also a hardware issue

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the packet-test failure in the Tofino Simulator using check_counter_incremented and the egress counter API call. Start with the P4 Egress control shown in the issue and compare its behavior with the working ingress-port counter case. Done means egress counters report the increments shown in simulator logs, the packet test passes, and the hardware comparison in the checklist is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.