Optimize memory consumption of take method in topsql stmtstats Aggregator
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
## Problem Description
Through pprof performance profiling, we discovered that the `aggregator.aggregate` function in the `pkg/util/topsql/stmtstats` package causes extremely high memory consumption when calling the `StatementStats.Take` method, which may make it a memory hotspot in the system.
## Performance Data
Based on pprof flame graph analysis:
### Memory Allocation Space (alloc_space)
- **Function**: `stmtstats.(*StatementStats).Take`
- **Memory Allocation**: 8.48TB (11% of total allocation)
- **Self Allocation**: 8.48TB (100% self-allocated, not from child function calls)
### Memory Allocation Object Count (alloc_objects)
- **Function**: `stmtstats.(*StatementStats).Take`
- **Object Allocation Count**: 194,524,634,791 (28.8% of total allocation)
- **Self Allocation**: 194,524,634,791 (100% self-allocated)
### CPU Time Consumption
- Overall CPU time of `stmtstats.(*aggregator).aggregate` function: 10.32s (4%)
- CPU overhead related to `stmtstats.(*StatementStats).Take` is also significant
## Call Stack
```
stmtstats.(*aggregator).run
└─ stmtstats.(*aggregator).aggregate
└─ sync.(*Map).Range
└─ stmtstats.(*aggregator).aggregate.func1
└─ stmtstats.(*StatementStats).Take ← Memory hotspot
```
## Impact
1. **Memory Pressure**: The `Take` method accounts for over 10% of total memory allocation, which may lead to increased memory pressure in high-concurrency scenarios
2. **GC Pressure**: Large amounts of object allocation will increase garbage collection frequency and duration, affecting overall performance
## Optimization Suggestions
1. **Object Pooling**: Consider using object pools (sync.Pool) to reuse frequently allocated objects in the `Take` method
2. **Batch Processing Optimization**: Check if batch processing logic in the `aggregate` function can be optimized to reduce memory allocation per `Take` call
Contributor guide
Assessment
This issue has not been assessed yet.