apache / apache/beam

[Feature Request]: Expand the info provided in Beam Python's heap dump

Open
#39,244 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Beam has a codepath to append heap information into sdk status reports:
[worker_status.py](https://github.com/apache/beam/blob/227bcded5aa2007cfb496378c876a75766f73b0d/sdks/python/apache_beam/runners/worker/worker_status.py#L111), enabled by the [--experiments=enable_heap_dump](https://github.com/apache/beam/blob/27d699eb7215ca4ebda8d61984c17507e2937e87/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L170) option, see [description](https://docs.google.com/document/d/12IBAxfGgLbAWxyB5hEHDIwm821FiTA7csd9PhBR3icI/edit?tab=t.0#heading=h.li0h8ahbwaj)

We can expand on the info we are currently collecting to allow more introspection into memory allocated by the process so that we can reason about whether memory is growing on the native heap vs Python allocations; we could also try to print glibc stats to provide insights into possible memory fragmentation, see: https://bloomberg.github.io/memray/memory.html#memory-can-be-fragmented for ideas.

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start with the heap-status path in sdks/python/apache_beam/runners/worker/worker_status.py and the enable_heap_dump option in sdk_worker_main.py. Review the linked description and Memray fragmentation guidance to define which native-heap, Python-allocation, and glibc statistics should be reported. Done means the enabled SDK status report exposes the agreed additional memory information for process introspection.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.