Azure / Azure/azure-sdk-tools

[RPaaS LiveValidation] Identify spike of heap memory usage in `validate-v2` image

Open
#6,023 0 comments 0 reactions 1 assignee Claimed by @scbedd View on GitHub
Central-EngSys
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

We saw the `node` process running within the `validate-v2` image _crash_ with error:

> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Then, because the docker doesn't automatically restart the node process, `liveness` and `readiness` k8 probes fail, and the pod is restarted.

We have temporarily addressed this by raising the node heap size to 2GB. This didn't necessitate an upgrade in actual memory allocation to each k8 pod, so it was a painless update.

However, the RPaaS team is a bit leery (rightfully so) of just expanding memory and ignoring this problem. We want to understand _why_ this is occurring so that we can have a mitigation in place BEFORE the world explodes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.