aws / aws/eks-anywhere

Upgrading eks anywhere baremetal cluster failed due to ipmi related issue

Open
#4,540 6 comments 0 reactions 0 assignees View on GitHub
area/providers/tinkerbell external team/providers
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**: Upgrading eks anywhere baremetal cluster failed due to ipmi related issue.
- error message of 'kubectl describe bmt -n eksa-system'
`Failed to connect to BMC: failed to open connection to BMC: 5 errors occurred:
* provider: gofish: Get "https://[ip]/redfish/v1/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
* provider: ipmitool: Error: Unable to establish IPMI v2 / RMCP+ session: exit status 1
* provider: *asrockrack.ASRockRack: Error logging in: Post "https://[ip]/api/session": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
* provider: idrac9: Post "https://[ip]/sysmgmt/2015/bmc/session": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
* no Opener implementations found`

**What you expected to happen**: upgrade normally

**How to reproduce it (as minimally and precisely as possible)**:
1. create k8s cluster with 2 machines and 5 hardwares -> pass
- hardware.csv containing 5 available hardwares
- cluster-mgmt-cluster.yaml file with 1 cp, 1dp specification

2. upgrade cluster -> fail
- cluster-mgmt-cluster-upgrade.yaml, file that is just different in k8s version from the file that is used to create cluster

**Anything else we need to know?**:
- Error occurred in both boot modes - uefi, bios
- hardware and idrac version

model | idrac version
-- | --
Dell PowerEdge R720 | 7
Dell PowerEdge R730 | 8
Dell PowerEdge R730 | 8
Dell PowerEdge R720 | 7
Dell PowerEdge R720 | 7

**Environment**:
- EKS Anywhere Release: 0.13.1
- EKS Distro Release: 1.23 -> 1.24
- hardware model and idrac version

Contributor guide

Open the contributing guide

Research direction

Start with the `kubectl describe bmt -n eksa-system` output and compare the create and upgrade cluster YAML files, especially the Kubernetes version change. Reproduce the upgrade against the listed Dell PowerEdge and iDRAC versions, then determine whether the BMC connection failure is resolved and the bare-metal cluster upgrades successfully.

Written by the indexing model from the issue text.

Assessment

Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.