BlueQuartzSoftware / BlueQuartzSoftware/simplnx
PERF: WritePoleFigure requires O(N) contiguous Euler buffer per phase
- Dominant language
- C++
- Stars
- 17
- Forks
- 13
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 10
Description
## Summary
`WritePoleFigureFilter` allocates a single contiguous `ebsdlib::FloatArrayType` buffer sized to all cells of the current phase before handing it to ebsdlib's pole-figure computation. On very large datasets this buffer dominates the filter's memory footprint because ebsdlib's pole-figure kernels require the full buffer up front — there is no streaming, chunked, or callback interface.
## Current behavior
In `src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.cpp` on `develop` (as of `9b667ae0b`), the per-phase loop around lines 565–609 does:
1. Walk every cell and count cells matching the current phase (+ optional mask) → `count`.
2. Allocate a flat buffer:
```cpp
const ebsdlib::FloatArrayType::Pointer subEulerAnglesPtr =
ebsdlib::FloatArrayType::CreateArray(count, eulerCompDim, "Euler_Angles_Per_Phase", true);
subEulerAnglesPtr->initializeWithValue(std::numeric_limits::signaling_NaN());
```
sized `count × 3 × sizeof(float)`.
3. Walk every cell again and densify matching Eulers into that buffer.
4. Pass the raw pointer through `PoleFigureConfiguration_t::eulers` to the downstream ebsdlib kernels:
```cpp
ebsdlib::PoleFigureConfiguration_t config;
config.eulers = subEulerAnglesPtr.get();
// ...
figures = makePoleFigures(config);
intensityImages = createIntensityPoleFigures(config, m_InputValues->NormalizeToMRD);
```
`makePoleFigures` and `createIntensityPoleFigures` (both defined in ebsdlib) walk `config.eulers` linearly. They don't expose a streaming, chunked, or callback interface, so the simplnx caller has to materialize the full buffer before invoking them.
## Impact
For a phase containing 500M cells, step 2 allocates ~6 GB. The peak heap is bounded by `max_over_phases(cells_in_phase) × 12 bytes`, regardless of whether the filter reads its inputs lazily or streams its output images. On datasets that approach or exceed available RAM, this allocation is the hard ceiling.
## Required changes
Resolving this requires a new API in `BlueQuartzSoftware/ebsdlib` first. Three candidate directions:
1. **Streaming pole-figure kernel** — add an overload of `makePoleFigures` / `createIntensityPoleFigures` that accepts a callback or iterator yielding `(count, const float* chunk)` pairs and incrementally updates the pole-figure bins.
2. **Memory-mapped backing for `PoleFigureConfiguration_t::eulers`** — allow the buffer to be backed by a memory-mapped file so the ebsdlib kernel's linear walk is served by file cache instead of anonymous RAM.
3. **Per-chunk accumulator** — expose the pole-figure bin buffers as a public accumulator type so callers iterate their own chunks of Eulers and call `accumulate(eulers_chunk)` then `finalize()`.
(1) and (3) are idiomatic. (2) is the lowest-effort if mmap semantics fit the access pattern.
Once the ebsdlib side exists, `WritePoleFigure.cpp` would be rewritten to iterate chunks of phase + Eulers directly into the new API, eliminating the `subEulerAnglesPtr` allocation entirely.
Contributor guide
Assessment
This issue has not been assessed yet.