apache / apache/cloudstack

VXLAN persistent networks create bridges on hosts that never ran a VM on them, but don't clean them up

Open
#13,966 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

With a persistent network offering, CloudStack builds the network's bridge on **every** host in the zone. But when the network is deleted it only cleans the bridge up on hosts that actually ran a VM on it. On every other host, the bridge and its VXLAN interface stay behind forever.

**Investigation**
There is a difference between the filters when creating a persistent network, and removing a persistent network.
**Creation:**
`DefaultHostListener.setupPersistentNetwork()` creates the bridges on hostConnect/hostEnabled for all networks from `getAllPersistentNetworksFromZone()` - no isolation-method filter.
**Removal:**
`NetworkOrchestrator.cleanupPersistentnNetworkResources()` is gated by `networkMeetsPersistenceCriteria()`, which requires the broadcast URI scheme to be `Vlan`, so for `vxlan://` networks `CleanupPersistentNetworkResourceCommand` is never sent. (`hostAboutToBeRemoved()` sends the same cleanup with no scheme check either - the VLAN-only gate exists only on the network-delete path.)

Hosts that ran a VM on the network are cleaned up via the normal VM-lifecycle teardown, so the leak only affects uninvolved hosts.

### versions

4.22.1.1, KVM, advanced zone with VXLAN isolation, persistent network offerings.

### The steps to reproduce the bug

1. Advanced zone, KVM, VXLAN isolation, 3+ hosts.
2. Create a VPC with two tiers on a persistent offering.
3. Start VMs so they land on only some hosts. (Restarting `cloudstack-agent` on an uninvolved host also creates the bridges there via hostConnect.)
4. Delete the tiers, then the VPC.
5. `ls -d /sys/class/net/brvx-*` on each host.

Expected behaviour:
The bridge and VXLAN interface are removed from every host in the zone.

Actual behaviour:
The VNI is still present on hosts that never ran a VM on the network; `CleanupPersistentNetworkResourceCommand` never appears in `management-server.log` during the delete.

### What to do about it?

Accept `Vxlan` alongside `Vlan` in `networkMeetsPersistenceCriteria()`, or drop the scheme check to match the setup path.

Contributor guide

Open the contributing guide

Research direction

Start with NetworkOrchestrator.cleanupPersistentnNetworkResources() and networkMeetsPersistenceCriteria(), then compare them with DefaultHostListener.setupPersistentNetwork() and hostAboutToBeRemoved(). Reproduce the VXLAN persistent-network deletion flow on three or more hosts and inspect management-server.log and /sys/class/net/brvx-*; done means cleanup is sent and bridges and VXLAN interfaces disappear from every host.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
infrastructure, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.