apache / apache/cloudstack

Redundant VPC VR goes dual PRIMARY (race condition?)

Open
#11,710 23 comments 1 reaction 0 assignees View on GitHub
component:virtual-router status:needs-investigation
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

When multiple network tiers are brought up in quick succession in a redundant VPC, I can consistently reproduce both VPCs going primary.

This is really bad because the VIP is then owned by both Virtual Routers and it means traffic is getting dropped like crazy, it makes the entire VPC unusable. Restarting the VPC recovers it to a good state.

Please see the comments for debugging information, but in short it appears at least one of the nodes ends up getting munged address information which causes the wrong ips to be programmed on interfaces and the keepalived configs to diverge between nodes which causes the dual primary. The underlying cause of the issue is this munged address information.

### versions

4.21.0

### The steps to reproduce the bug

Use Terraform to create the VPC, network tiers, and VM Instances.
Terraform configuration being used is here: https://github.com/bradh352/terraform-config

### What to do about it?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the dual-primary state using the Terraform configuration linked in the issue, with the VPC, network tiers, and VM instances on CloudStack 4.21.0. Read the debugging information in the comments and trace the munged address information through interface programming and the divergent keepalived configurations. Done means rapid network-tier creation no longer produces two primary VPCs or a shared VIP, and the recovery behavior is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
terraform
Domain
cloud, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.