[Feature Request]: Expand the info provided in Beam Python's heap dump
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
Beam has a codepath to append heap information into sdk status reports:
[worker_status.py](https://github.com/apache/beam/blob/227bcded5aa2007cfb496378c876a75766f73b0d/sdks/python/apache_beam/runners/worker/worker_status.py#L111), enabled by the [--experiments=enable_heap_dump](https://github.com/apache/beam/blob/27d699eb7215ca4ebda8d61984c17507e2937e87/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L170) option, see [description](https://docs.google.com/document/d/12IBAxfGgLbAWxyB5hEHDIwm821FiTA7csd9PhBR3icI/edit?tab=t.0#heading=h.li0h8ahbwaj)
We can expand on the info we are currently collecting to allow more introspection into memory allocated by the process so that we can reason about whether memory is growing on the native heap vs Python allocations; we could also try to print glibc stats to provide insights into possible memory fragmentation, see: https://bloomberg.github.io/memray/memory.html#memory-can-be-fragmented for ideas.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Start with the heap-status path in sdks/python/apache_beam/runners/worker/worker_status.py and the enable_heap_dump option in sdk_worker_main.py. Review the linked description and Memray fragmentation guidance to define which native-heap, Python-allocation, and glibc statistics should be reported. Done means the enabled SDK status report exposes the agreed additional memory information for process introspection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100