NVIDIA / NVIDIA/cudf

[FEA] add debug log for HostMemoryBuffer allocation

Open
#17,444 2 comments 0 reactions 0 assignees View on GitHub
feature request Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

When we're tracing some container OOM issues (exit code 137, killed by K8S because too much memory usage) of Spark Rapids, we find most of executor memory is allocated via HostMemoryBuffer. Meanwhile, after @revans2 's https://github.com/rapidsai/cudf/pull/17197, in theory the total amount of memory allocated by HostMemoryBuffer will be bounded by `spark.rapids.memory.host.offHeapLimit.size`.

It is nice and important if we can have a metrics (like a Gauge in promethus) telling us how much memory is consumed by HostMemoryBuffer. Since we don't have embedded monitoring solution (like promethus+grafana), we'd better log HostMemoryBuffer usage each time it reaches a new watermark.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.