astropy / astropy/reproject

Reduce memory usage of pixel_to_pixel

Open
#534 4 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
127
Forks
74
Avg merge
1d 11h
Merged PRs (30d)
2

Description

This isn't really an issue that requires work in reproject but it might require work in several other packages. Basically pixel_to_pixel can be very memory-intensive under certain situations. Here's an example:

https://gist.github.com/astrofrog/a807da0c34b7d669289e3c14d43a099a

this is with a GWCS and regular astropy WCS object. The input arrays are 80Mb each (160Mb total) but the memory peaks around 1.7-1.8Gb which is 10x more than this. It might be there's not much we can do about it, but it'd be interesting to see if there are places where we memory could be used more efficiently. Obviously, the output arrays also need 80Mb each, but that's still a lot of overhead.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the pixel_to_pixel entry point and reproduce the memory profile using the linked gist with GWCS and astropy WCS inputs. Trace where the 80 MB input and output arrays are duplicated, then compare peak memory and output correctness after any change; the issue does not name specific files or a target peak.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.