aws / aws/eks-anywhere

Uploading bottlerocket images to vsphere fails

Open
#7,286 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
Uploading bottlerocket images to vsphere fails.

**What you expected to happen**:
New bottlerocket images should be uploaded to vsphere.

**How to reproduce it (as minimally and precisely as possible)**:
When upgrading an existing management cluster, eksctl anywhere is not able to automatically upload the bottlerocket image to vsphere.

**Anything else we need to know?**:
```
V1 Worker node VSphereMachineConfig template is not set. Using default template.
V5 Retrier: {"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=xxx -e GOVC_INSECURE=false -e GOVC_DATACENTER=xxx -e GOVC_TLS_KNOWN_HOSTS=xxx/generated/govc_known_hosts eksa_1704982472982975015 govc find -json /xxx -type VirtualMachine -name bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2"}
V5 Retry execution successful {"retries": 1, "duration": "224.185878ms"}
V2 Template not found: /xxx/vm/Templates/bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2
V2 Template not available. Creating {"template": "/xxx/vm/Templates/bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2"}
V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=xxx -e GOVC_INSECURE=false -e GOVC_DATACENTER=xxx -e GOVC_TLS_KNOWN_HOSTS=xxx/generated/govc_known_hosts eksa_1704982472982975015 govc library.ls eks-a-templates"}
V0 Creating template. This might take a while.
V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_TLS_KNOWN_HOSTS=xxx/generated/govc_known_hosts -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=xxx -e GOVC_INSECURE=false -e GOVC_DATACENTER=xxx eksa_1704982472982975015 govc library.info -json eks-a-templates/bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2"}
V2 Importing template from ova url {"ova": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/54/artifacts/ova/1-28/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova"}
V4 Importing template {"ova": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/54/artifacts/ova/1-28/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova", "templateName": "bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2"}
V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=xxx -e GOVC_INSECURE=false -e GOVC_DATACENTER=xxx -e GOVC_TLS_KNOWN_HOSTS=xxx/generated/govc_known_hosts eksa_1704982472982975015 govc library.import -k -pull -n bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2 eks-a-templates https://anywhere-assets.eks.amazonaws.com/releases/bundles/54/artifacts/ova/1-28/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova"}
V9 docker {"stderr": "govc: The import of library item 5a686415-66d7-43d0-9b93-ab7a64f1cea4 has failed. Reason: Error transferring file bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova to ds:///vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f//contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64_86c22c6f-777c-43fc-8c17-18daa21a926d.ova?serverId=a662a392-9aea-4653-8698-d78db059ddab. Reason: Error during transfer of ds:///vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f//contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64_86c22c6f-777c-43fc-8c17-18daa21a926d.ova?serverId=a662a392-9aea-4653-8698-d78db059ddab: IO error during transfer of ds:/vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f/contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-vmware-k8s-1.28-x86_64-1.15.1-264e294c_86c22c6f-777c-43fc-8c17-18daa21a926d.vmdk: Pipe closed.\n"}
V0 ❌ Validation failed {"validation": "vsphere provider validation", "error": "failed setting default values for vsphere machine configs: failed importing template into library: importing template: govc: The import of library item 5a686415-66d7-43d0-9b93-ab7a64f1cea4 has failed. Reason: Error transferring file bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova to ds:///vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f//contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64_86c22c6f-777c-43fc-8c17-18daa21a926d.ova?serverId=a662a392-9aea-4653-8698-d78db059ddab. Reason: Error during transfer of ds:///vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f//contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64_86c22c6f-777c-43fc-8c17-18daa21a926d.ova?serverId=a662a392-9aea-4653-8698-d78db059ddab: IO error during transfer of ds:/vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f/contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-vmware-k8s-1.28-x86_64-1.15.1-264e294c_86c22c6f-777c-43fc-8c17-18daa21a926d.vmdk: Pipe closed.\n", "remediation": ""}
```

We had a support call with vmware, where an engineer found an internal error report, which says that the vmdk file of the bottlerocket image is filled with zeros at the end, which the NFC (Network File Copy) service in ESX removes as an optimization, whereby the hash of the vmdk no longer fits and the vcenter makes a rollback and deletes the file. This problem is supposed to be fixed with vsphere 8, but according to him it will no longer be patched for 7.x.

**Environment**:
eksctl version:
0.167.0
eksctl anywhere version:
v0.18.3
vsphere version:
7.0.3

Contributor guide

Open the contributing guide

Research direction

Start at the vSphere provider validation and its default machine-config/template import path; use the logged govc library.import failure on vSphere 7.0.3 as the reproduction point. Confirm the handling of the Bottlerocket OVA/VMDK transfer, with done defined as a management-cluster upgrade successfully uploading the image.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.