Improve heap dump creation for customer instances

Problem Statement

Customer instances need better heap dump creation capabilities for troubleshooting OutOfMemoryError issues and on-demand memory analysis.

Requirements

1. Automatic heap dump on OOM: Enable -XX:+HeapDumpOnOutOfMemoryError JVM flag
2. On-demand heap dump creation: Provide mechanism for customers/support to trigger heap dumps manually
3. Standardized storage location: Store heap dumps in /data/shared folder structure for consistent access

Implementation Details

Automatic OOM Heap Dumps

• Configure JVM startup with -XX:+HeapDumpOnOutOfMemoryError
• Set heap dump path to /data/shared/heapdumps/ directory
• Use -XX:HeapDumpPath=/data/shared/heapdumps/

On-Demand Heap Dump Creation

• Create REST endpoint or admin utility for manual heap dump generation
• Use JMX MBeans (HotSpotDiagnosticMXBean.dumpHeap()) for programmatic dumps
• Include timestamp and instance identifier in filename

Directory Structure

/data/shared/
├── heapdumps/
│   ├── oom-dump-[timestamp]-[pid].hprof
│   └── manual-dump-[timestamp]-[instance].hprof

Acceptance Criteria

• JVM automatically creates heap dump on OutOfMemoryError
• Heap dumps are stored in /data/shared/heapdumps/ directory
• On-demand heap dump creation is available via admin interface
• Heap dump files include timestamp and identifying information
• Directory permissions allow customer access to dump files
• Documentation updated with heap dump procedures

Technical Considerations

• Ensure sufficient disk space monitoring for heap dumps
• Consider heap dump file rotation/cleanup policies
• Add proper error handling for dump creation failures
• Include heap dump creation in monitoring/alerting

Start by locating the JVM startup configuration and the admin interface, then review the listed -XX heap-dump flags and HotSpotDiagnosticMXBean.dumpHeap() approach. Done means automatic and on-demand dumps are available in /data/shared/heapdumps/, filenames identify the dump, permissions and failure handling work, and the heap-dump procedures are documented.