Consider using single-precision
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 3
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 17
Description
Currently we mostly use double precision (float64) for everything in the data reduction. This affects event weights as well as coordinates.
- Coordinates such as
wavelength,Q,two_theta,phi,Qx,Qy, ... all default to float64. This can consume a lot of memory, and thus makes coord transform as well as all binning/grouping operations slower. We should tests if switching to float32 negatively impacts there results. Are there any operations that would be highly sensitive to the precision? - For event weights I feel there are more pitfalls with float32, since they end up getting summed and precision can easily get lost here, i.e., that may require a lot more care.
As a quick test, I converted tof to float32 right after loading. This saves about 3 seconds for 1e9 events (total reduction time 50 seconds, i.e., 5% savings). The effect may be larger if gravity is enabled, since the coord transforms get more complex.
Note: If we plan to do this, we should actually push the conversion into the loader, e.g., by providing a customization of the NXevent_data application definition to ScippNexus. This is relevant since ScippNexus performs binning into pixels which would already see performance benefits from smaller elements.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the data-reduction paths for coordinates and event weights, then inspect the ScippNexus NXevent_data application definition and the existing tof conversion after loading. Compare float32 and float64 for reduction results, memory use, and timing, including gravity-enabled coordinate transforms and binning; document whether the precision trade-offs support changing the loader.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100