aws / aws/eks-distro

EKS v1.30.4 Upgrade Caused Instant Service CrashbackLoop

Open
#3,370 4 comments 3 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
1.5k
Forks
198
Avg merge
10h 46m
Merged PRs (30d)
28

Description

**What happened**:

Our AWS infrastructure upgraded our EKS nodes from `v1.30.1-eks-e564799` to `v1.30.4-eks-16b398d`. The instant this happened, all of our production services crashed, all at once. With the exception of one pod on a few of our services where minimum node tolerations were configured.

Every single one of our (NodeJS and Java) services fell into a crashback loop all at once.

If true, it could be traced to this release https://github.com/aws/eks-distro/releases/tag/v1-30-eks-15

NodeJS Error
```
#
# Fatal error in , line 0
# Check failed: 12 == (*__errno_location ()).
#
#FailureMessage Object: 0xffffee1205a0
1: 0xceb064 [node]
2: 0x1f43eb0 V8_Fatal(char const*, ...) [node]
3: 0x1f4e5e8 v8::base::OS::SetPermissions(void*, unsigned long, v8::base::OS::MemoryPermission) [node]
4: 0x10e1974 v8::internal::MemoryAllocator::SetPermissionsOnExecutableMemoryChunk(v8::internal::VirtualMemory*, unsigned long, unsigned long, unsigned long) [node]
5: 0x10e1cb4 v8::internal::MemoryAllocator::AllocateAlignedMemory(unsigned long, unsigned long, unsigned long, v8::internal::AllocationSpace, v8::internal::Executability, void*, v8::internal::VirtualMemory*) [node]
6: 0x10e1eb8 v8::internal::MemoryAllocator::AllocateUninitializedChunkAt(v8::internal::BaseSpace*, unsigned long, v8::internal::Executability, unsigned long, v8::internal::PageSize) [node]
7: 0x10e2488 v8::internal::MemoryAllocator::AllocatePage(v8::internal::MemoryAllocator::AllocationMode, v8::internal::Space*, v8::internal::Executability) [node]
8: 0x10f6e78 v8::internal::PagedSpaceBase::TryExpandImpl() [node]
9: 0x10f98c0 [node]
10: 0x10f9e54 v8::internal::PagedSpaceBase::RefillLabMain(int, v8::internal::AllocationOrigin) [node]
11: 0x1070988 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
12: 0x1050230 v8::internal::Factory::CodeBuilder::AllocateInstructionStream(bool) [node]
13: 0x1050604 v8::internal::Factory::CodeBuilder::BuildInternal(bool) [node]
14: 0xed068c v8::internal::baseline::BaselineCompiler::Build(v8::internal::LocalIsolate*) [node]
15: 0xee2a04 v8::internal::GenerateBaselineCode(v8::internal::Isolate*, v8::internal::Handle) [node]
16: 0xf3c1b0 v8::internal::Compiler::CompileSharedWithBaseline(v8::internal::Isolate*, v8::internal::Handle, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
17: 0xf3c734 v8::internal::Compiler::CompileBaseline(v8::internal::Isolate*, v8::internal::Handle, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
18: 0xece3cc v8::internal::baseline::BaselineBatchCompiler::CompileBatch(v8::internal::Handle) [node]
19: 0xf4708c v8::internal::Compiler::Compile(v8::internal::Isolate*, v8::internal::Handle, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
20: 0x14587a8 v8::internal::Runtime_CompileLazy(int, unsigned long*, v8::internal::Isolate*) [node]
21: 0x1862a84 [node]
```

The Java error was less verbose but looks like
```
Error occurred during initialization of VM
Failed to mark memory page as executable - check if grsecurity/PaX is enabled
```

**What you expected to happen**:

Auto scheduled upgrade to patch version bump `1.30.4` does not cause all services to crash with internal Memory errors.

**How to reproduce it (as minimally and precisely as possible)**:

I don't currently have a reproduction, I want this bug report to represent a centralised location for others who potentially experienced the same issue to engage in conversation here. I also cannot guarantee with certainty its an issue with the EKS image, but it does seem very suspicious that the upgrade to 1.30.4 caused it, and downgrade fixed it instantly.

**Anything else we need to know?**:

**Environment**:
- EKS Distro Release Channel: https://github.com/aws/eks-distro/releases/tag/v1-30-eks-15
- EKS Distro Release: `v1.30.4-eks-16b398d`

**Related**
- Posted at the same time (https://github.com/nodejs/node/issues/55509)
- Older post (https://github.com/nodejs/help/issues/4465)

Contributor guide

Open the contributing guide

Research direction

Start by comparing the v1.30.1 and v1.30.4 EKS-D release details, then review the linked Node.js issue and older help report for matching executable-memory failures. A useful resolution would identify whether the EKS image release caused the Node.js and Java crashes and provide a reproducible confirmation or a documented cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, kubernetes, node.js
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.