aws / aws/containers-roadmap

[EKS NodeGroups] [VersionUpdate]: VersionUpdate needs more information on errors and runtime

Open
#2,522 0 comments 1 reaction 0 assignees View on GitHub
EKS EKS Managed Nodes Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

# Tell us about your request
VersionUpdate has a start time, it also needs an end time

NodeCreationFailure and PodEvictionFailure are ambiguous with several potential root causes, there should be a more detailed message when VersionUpdate goes to an error state to aid diagnosis.

Errors in VersionUpdate should be time stamped.

## Which service(s) is this request for?
AWS EKS

## Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
VersionUpdate can fail for many reasons as documented below however these appear to fall into NodeCreationFailure or PodEvictionFailure
https://docs.aws.amazon.com/eks/latest/userguide/managed-node-update-behavior.html

It is very difficult to establish the root cause, very difficult to establish the timing of errors and total runtime for a VersionUpdate.

VersionUpdate does have a start time but the errors are not timestamped which is critical if attempting to diagnose problems.

VersionUpdate does not have an end time with makes it difficult to establish the runtime for updates, something which is important when we are measuring maintenance windows.

As per the documentation for NodeCreationFailure and PodEvictionFailure has several possible causes which are not explicitly mentioned in the Errors.

It is important for AWS EKS to report the decision for moving VersionUpdate to an Error status with as much precision as possible to assist troubleshooting efforts.

For instance I have spent a long time working with AWS support only to identify that the total runtime for each new node is capped at 15-minutes, this is not mentioned as a possible cause of NodeCreationFailure and is not recorded in the error as the root cause. Out of the 4 possible errors, [node bootstrap taking more than 15-minutes is a 5th undocumented root cause](https://github.com/awsdocs/amazon-eks-user-guide/issues/842).

Couple this with a lack of timing information and an issue which is caused by nodes laking more than 15-minutes to bootstrap is very difficult to diagnose.

## Are you currently working around this issue?
It is necessary to watch the node creation carefully if we begin to encounter errors, the root cause can only be gleaned by catching it in realtime or conducting a lengthy and costly investigation using CloudTrail.

CloudTrail does not record the end time for VersionUpdate

## Additional context
Feedback was given to the authors of the documentation page below asking them to add the 15-minute note creation time limit as another root cause for NodeCreationFailure and make it clear that the 15-minutes appears to include the time for EKS to initiate scale-out, EC2 instance bootstrap, custom UserData and the time taken to run the script which connects the EC2 instance as an EKS node.

https://docs.aws.amazon.com/eks/latest/userguide/managed-node-update-behavior.html

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the AWS EKS managed node update behavior documentation and the linked amazon-eks-user-guide issue 842. Compare the documented VersionUpdate behavior with the requested error details, timestamps, and end time. Done means the VersionUpdate result exposes precise failure causes, error timing, and total runtime for diagnosis.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.