envoyproxy / envoyproxy/gateway

Flaky E2E Test: UDPRouteBackendFQDNTest/UDPRoute_with_a_FQDN_type_Backend

Open
#9,360 3 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

## Problem

The E2E test `TestE2E/UDPRouteBackendFQDNTest/UDPRoute_with_a_FQDN_type_Backend` is flaky and fails intermittently with UDP DNS query timeouts.

## Evidence

### Recent Failures

1. **PR #9359**: `e2e-test (v1.34.8, ipv4, default)` - FAIL
- Error: `failed to perform a UDP query: read udp 172.18.0.1:58505->172.18.0.206:5300: i/o timeout`
- Run: https://github.com/envoyproxy/gateway/actions/runs/28210980762/job/83573440003

2. The same test passes in other configurations:
- v1.33.12 ipv4 - PASS
- v1.35.5 ipv6 - PASS
- v1.36.1 all configs - PASS

3. On main branch recent runs, `e2e-test (v1.34.8, ipv4, default)` passes:
- Run: https://github.com/envoyproxy/gateway/actions/runs/28210631971

### Pattern

- Fails only on **Kubernetes v1.34.8 + IPv4** configuration
- Failure is always **UDP DNS timeout** (not a code issue)
- Passes on other K8s versions and IPv6
- Passes when rerun

## Root Cause

This appears to be an infrastructure/CI environment issue:
1. DNS resolution timeout in UDP queries
2. Specific to K8s v1.34.8 cluster setup
3. Network/DNS configuration issue in the test environment

## Impact

This flaky test blocks legitimate PRs from passing CI, requiring manual retest or approval.

## Suggested Fixes

1. **Immediate**: Increase UDP timeout in the test (currently appears to be 2s, could increase to 5s)
2. **Investigation**: Check DNS/CoreDNS configuration in v1.34.8 cluster setup
3. **Long-term**: Consider making the test more resilient to network delays

## Test Location

- File: `test/e2e/tests/udproute_with_backend.go:96`
- Test: `UDPRouteBackendFQDNTest`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with test/e2e/tests/udproute_with_backend.go:96 and the UDP query timeout used by UDPRouteBackendFQDNTest. Reproduce the Kubernetes v1.34.8 IPv4/default E2E configuration and compare its DNS or CoreDNS behavior with passing configurations. Done means the test no longer flakes in that configuration without masking genuine failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
ci-cd, networking, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.