astropy / astropy/specutils

Assess handling of large cubes

Open
#747 0 comments 0 reactions 0 assignees View on GitHub
spectral-cube
Dominant language
Python
Stars
204
Forks
134
Avg merge
12h 55m
Merged PRs (30d)
1

Description

This is probably a task that should wait until some of the other `spectral-cube` transition work is done. Essentially, `spectral-cube` takes some care in handling large cubes, in terms of avoiding copying data and providing hints on how to use memory mapping (from fits files) to handle cubes that are too big to fit in memory. It's possible that `specutils` handling of large data is already fine, but it would be prudent to do a side-by-side test with `spectral-cube` of a large dataset and see if some effort to improve large dataset handling/port over some strategies from `spectral-cube` would be prudent.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing specutils and spectral-cube handling of a large dataset side by side. Check whether specutils avoids unnecessary copies and supports memory mapping for FITS data that exceeds available memory. Done means documenting the comparison and determining whether specific handling strategies should be ported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.