astropy / astropy/ccdproc

Tracking: remaining work for the array-API migration

Open
#971 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
93
Forks
92
Avg merge
14h 44m
Merged PRs (30d)
30

Description

This is the canonical checklist for finishing the array-API (array-library-agnostic) migration. It supersedes #909 as the place to track status; #909 and its children (#910–#913) remain as the CuPy-specific findings.

**State as of 2026-08-25 (post-#989/#990):** the August batch (#972–#980, plus #985 CI coverage), #987 (NaN-aware sum/mean/std/median fallbacks, closing #986), #988 (combiner internals: `xp.any`/`xp.count_nonzero`, namespace bool mask dtypes, `nbytes`-free memory sizing), #989 (array-API `median` fallback for `subtract_overscan`, wrapping `nanmedian` with numpy's NaN-propagating semantics) and #990 (test-only hygiene batch addressing #969/#970 and the test-body numpy-isms) are all merged. #992 (combine-method `mask=` bucket, 63 → 54) and #993 (`gain_correct`/`flat_correct` `device=`, 54 → 41) are merged too. #995 (namespace dtypes: `ccd_process` bool, `transform_image` mask, `combine()`/`ImageFileCollection` FITS byte order) and #994 (`clip_extrema` rank-comparison rewrite) are merged too (2026-08-25, `6724c8e`): strict is now **31** on `main`, and every section-1 bucket that was on the strict failure list is closed. What is left on strict is 18 test-body fixes (section 2) and 13 upstream-blocked tests (section 3). CI exercises numpy, jax, dask (including dask+bottleneck and a dask escape-baseline gate); `array-api-strict` runs as a soft (`continue-on-error`) job. CuPy and torch are not in CI. User docs live in `docs/array_api.rst`. 36 `backend_xfail` markers remain in the test suite. **Later on 2026-08-25, three PRs are open on top of `6724c8e`:** #996 (narrows the two stale `test_cosmicray.py` markers, 5 XPASS → 0), #997 (#982 and the correlated add/subtract leak in the wrapper) and #998 (the section-2 test bodies, strict 31 → 15). Merged together locally they give strict **15 failed, 496 passed, 36 skipped, 45 xfailed, 0 xpassed**; numpy 557 passed / 35 skipped; jax and dask green apart from xfails. Of the 15, 11 are #929, one each is #936, #983 (`test_rebin_ccddata[True-True]`) and the two new ccdproc items below (`combine(output_file=)` writing a namespace array, `combine(dtype=int)`). **All three merged 2026-08-25 (`main` = `4d7d21b`); strict on `main` verified at exactly those 15.** #999 then took the two ccdproc items plus the `len(oscan)` one-liner; **merged 2026-08-26 (`main` = `be80055`)**. Strict on `main` verified at **13 failed / 503 passed / 36 skipped / 45 xfailed / 0 xpassed**, and every remaining failure is upstream-blocked: 8 × #929 via the default `sigma_func` (`median_absolute_deviation`), 3 × #929 via `Combiner.sigma_clipping` (`sigma_clip`), #936, #983. **Section 1 is complete.** #946 and #943 are closed. **2026-08-26, later:** the #929 half that is fixable on the ccdproc side is now in two PRs off `be80055`: #1000 (`sigma_func` → xp-native MAD when the namespace is not numpy; strict 13 → 5) and #1001 (`Combiner.sigma_clipping` → xp-native clip reproducing `astropy.stats.sigma_clip`; strict 13 → 10). Numpy keeps astropy in both. Merged together locally: strict **2 failed / 1129 passed / 0 xpassed** (#936, #983 only), the two `BOUNDARY` lines leave the escape baseline, and the dask enforce run is clean.

**State as of 2026-08-30 (post-#1005):** #1000, #1001 (with #1003's numpy>=2.0/astropy>=6.1 dependency bump) and #1005 are merged. Strict on `main` (`1bd015c`) is **0 failed / 839 passed / 10 skipped / 46 xfailed / 0 xpassed**, and the `py313-strict` job now runs in the regular CI matrix (the `continue-on-error` carve-out and `strict_status.yml` are removed; a strict failure fails CI). The two astropy-blocked tests carry `backend_xfail` markers citing #936 and #983. The escape baseline is 10 entries, all `BOUNDARY`; regenerating it from a full dask run is byte-identical and the enforce ratchet holds on dask and jax. Full-suite results per backend: numpy 890 passed; dask 879 passed (ratchet enforced); jax 885 passed (ratchet enforced). All 30 remaining `backend_xfail` markers were re-triaged 2026-08-30: every one is a genuine numpy-only boundary (astroscrappy, scipy.ndimage, reproject, astropy modeling/nddata/units).

Current strict result on `main` (1b9b621): **63 failed, 418 passed, 36 skipped, 46 xfailed, 5 xpassed** — down from 85 with #989 + #990. With #993 (`711bb26`) the strict job was 41; #995 and #994 (`6724c8e`) take it to **31 failed, 469 passed, 36 skipped, 43 xfailed, 5 xpassed** with no new failures; the 5 XPASSes are still the stale `test_cosmicray.py` markers. The job is `continue-on-error`, so read the pytest summary line in its log rather than the job conclusion. Every one of the 63 has been traced to a cause (2026-08-25):

| # | cause | where | section |
|---|---|---|---|
| 12 | `average_combine`/`sum_combine`/`median_combine` build the result with `CCDData(..., mask=mask)`; astropy's `NDData.mask` setter forces numpy and cannot convert a `device1` strict array | `combiner.py:726`, `:804`, `:616` | 1 |
| 9 | `gain_correct` calls `xp.asarray(gain_value)` without `device=` ("two different devices") | `core.py:968` | 1 |
| 2 | `flat_correct` calls `xp.asarray(flat_mean)` without `device=` (same class) | `core.py:1057` | 1 |
| 4 | `ccd_process` bad-pixel mask uses builtin `bool` instead of `xp.bool` | `core.py:383` | 1 |
| 4 | `clip_extrema` fancy-index assignment | `combiner.py:406` | 1 |
| 2 | `combine()` passes a numpy dtype (`ccd.data.dtype.type`) to the requested namespace's `asarray` — strict warns, `filterwarnings=error` fails | `combiner.py:1082` | 1 |
| 2 | uncertainty propagation for multiply/divide delegates to astropy's `_propagate_multiply_divide`, whose `np.abs(...)` (`nduncertainty.py:832`) densifies the strict array and the mixed expression then fails | `_ccddata_wrapper_for_array_api.py:305–330` | 1 |
| 1 | `transform_image` applies the user transform to the **bool** mask; strict rejects `10 * bool_array` (numpy silently promotes) | `core.py:1153` | 1 |
| 8 | `astropy.stats` densification: default `sigma_func` (`median_absolute_deviation`, 5) and `sigma_clipping` (`sigma_clip`, 3) | `core.py:1291`, `combiner.py:491` | 3 (#929) |
| 6 | test bodies call astropy `CCDData.multiply(...)` on strict data to build scaled inputs; astropy's `_prepare_then_do_arithmetic` does `np.result_type(ref, operand)` on a strict dtype ("Could not convert Array … to a NumPy dtype") — this is the NDData-arithmetic upstream item already in #940, not a ccdproc code path | `test_combiner.py` (`test_combiner_with_scaling*`, `test_combiner_result_dtype`, `test_combine_overwrite_output`) | 2 |
| 5 | `combine()` given FITS filenames / numpy `CCDData` with no `array_package=` returns numpy data, and the tests then compare it with strict arrays via `xp.all(xpx.isclose(...))`; `array_package` is opt-in by design, so the tests should pass it | `test_combine_average_fitsimages`, `test_combine_numpyndarray`, `test_combine_average_ccddata`, `test_combine_limitedmem_fitsimages`, `test_combine_limitedmem_scale_fitsimages` | 2 |
| 2 | test bodies `ccd.write()` a `device1` strict `CCDData`; `astropy.io.fits` cannot coerce it | `test_combiner_image_file_collection_input`, `test_combine_image_file_collection_input` | 2 |
| 2 | test body `xp.asarray(np_mgrid[...]) / 10.0` — integer strict array divided by a float | `test_ccdproc.py:304` (`test_subtract_overscan_model`) | 2 |
| 1 | test helper `_make_mean_scaler` uses `.mean()` / `np.ma.average` — **this is the "`Combiner.scaling` setter bug" from #990's deferred list; the setter is fine** | `test_combiner.py:52` (`test_3d_combiner_with_scaling`) | 2 |
| 1 | astropy `_arithmetic` decorator leaks `_config_ccd_requires_unit=False` (astropy/astropy#20268) | `test_generator_ccds_without_unit` | 2/3 |
| 1 | astropy `NDData.mask` setter (#983's blocker) | `test_rebin_ccddata[True-True]` | 2/3 |
| 1 | `Quantity` machinery on a `device1` strict array | `test_unit_mismatch_behaves_as_expected` | 3 (#936) |

So: **36 are ccdproc bugs fixable now** (section 1), **17 are test-body fixes** (section 2), and **10 are upstream** (#929 ×8, #936, astropy#20268) — plus the rebin/mask-setter one that is fixable with the `_mask` workaround if we want it. The 5 xpassed are all in `test_cosmicray.py`: the two `test_cosmicray_median_masked_column[masked_array-*]` cases (stale since #979) and the three `gain_apply=False` cases of `test_cosmicray_gain_correct_uncertainty_namespace` (the `gain_apply=True` cases still xfail, so that marker needs narrowing to the `True` parametrizations, not removing).

### 1. Fixable-now bugs in ccdproc (no upstream dependency)
- [x] #965 `Combiner.__init__` builds a nested `xp.asarray([...])`; needs `xp.stack` (44 strict failures — biggest single win)
- [x] #966 `subtract_dark(scale=True)` device-propagation bug
- [x] #967 `rebin()` uses the `.astype()` method instead of `xp.astype`
- [x] #968 `create_deviation` float × bool-mask multiplication (`core.py:459`)
- [x] #963 `background_deviation_box` discards the `xpx.at` result (silent no-op on immutable backends)
- [x] #962 Variance/InverseVariance uncertainty wrappers escape to numpy on jax (only StdDev is array-API-aware)
- [x] #932 dead code in `cosmicray_median` clobbers masked-input handling
- [x] #906 xp-native fallback median for backends without `scipy.ndimage`
- [x] #986 no array-API fallback for NaN-aware mean/sum/std -- fixed in #987, which also merged the #906 median fallback into `ccdproc/_nanfuncs.py`
- [x] `subtract_overscan` calls `xp.median` (`core.py:629`), which is not in the array API standard — fixed in #989 with a `median` fallback in `_nanfuncs.py`
- [x] `Combiner.average_combine`/`sum_combine`/`median_combine` pass `mask=` to the `CCDData(...)` constructor (`combiner.py:726`, `:804`, `:616`), so astropy's numpy-forcing `mask` setter runs on the result. `ccd_process` already sidesteps the same setter by assigning `nccd._mask` (`core.py:381`); do the same here. **12 strict failures — biggest remaining bucket** — **PR #992** (also fixes the same setter sites inside `combine()` and its numpy-only `mask.copy()`; the remaining members of the bucket now stop at #929 (`median_combine`'s default `sigma_func`) or at `test_writeable_after_combine`, which writes a `device1` array to FITS and belongs with the section-2 "write a numpy copy" tests)
- [x] `gain_correct` calls `xp.asarray(gain_value)` without `device=` (`core.py:968`), so the scalar lands on the default device and the multiply fails when the image is elsewhere. Surfaced by #989; **9 strict failures** (`test_gain.py` ×5, `test_gain_correct*` ×3, `test_ccd_process_does_not_change_input`) — **PR #993** (also makes a plain-number gain a `float`, since the standard does not promote `int64` with `float64`)
- [x] `flat_correct` has the same bug for `xp.asarray(flat_mean)` (`core.py:1057`). 2 strict failures (`test_flat_correct`, `test_flat_correct_norm_value`) — **PR #993** (the two tests then needed their flats created on the testing device; fixed in the same PR)
- [x] `ccd_process` builds its bad-pixel mask with `xp.asarray(mask, dtype=bool)` — Python builtin `bool` where `xp.bool` is required (`core.py:383`; same class as #968/#988's fixes). Surfaced by #989; 4 strict failures — **PR #995** (the same four tests then hit `log_meta._replace_array_with_placeholder`, which assumed anything without `__len__` is NDData and raised `AttributeError` on a strict array; fixed in the same PR, plus the test bodies now build a bool mask and compare with `==`)
- [x] `Combiner.clip_extrema` assigns through fancy indexing (`combiner.py:406`, `IndexError: Fancy indexing __setitem__ is not supported`); needs `xpx.at` or a `where`-based rewrite. 4 strict failures — **PR #994** (no scatter at all: `argsort` twice gives each image's per-pixel rank, and `ranks < nlow | ranks >= n - nhigh` is the mask; bit-identical to the old code on 8400 random trials incl. ties/NaNs)
- [x] `combine()` converts file-read data with `xp.asarray(ccd.data, dtype=ccd.data.dtype.type)` (`combiner.py:1082`, and the uncertainty line below it), handing a numpy dtype to the target namespace; strict warns and the suite's `filterwarnings = error` turns it into a failure. Let `xp.asarray` infer the dtype or map it through the namespace. 2 strict failures (`test_combine_ccd_with_uncertainty_and_mask_from_fits`) — **PR #995** (dropping `dtype=` alone is not enough: strict and jax reject big-endian `>f8` outright, so a `_native_numpy` helper converts to native byte order in numpy first; the same three sites in `ImageFileCollection.ccds`/`data` get the same fix, and its mask has to go through `_mask` because the public setter forces numpy). Only `[mean]` clears; `[function]` then stops at `_make_mean_scaler` (section 2)
- [x] `_CCDDataWrapperForArrayAPI` uncertainty: `_propagate_multiply`/`_propagate_divide` call `super()._propagate_multiply_divide`, and astropy's implementation uses `np.abs(...)` (`nduncertainty.py:832`), which densifies the strict operand; the wrapper needs to own that formula the way it owns add/subtract (upstream fix would be `abs()` instead of `np.abs`; worth a note in #940). 2 strict failures (`test_flat_correct_deviation`, `test_flat_correct_data_uncertainty`) — **PR #993** (the mixin now owns `_propagate_multiply_divide`; after the `device=` fix this bucket also sat behind the five `test_gain.py` failures; the two strict xfail markers in `test_ccddata_wrapper_for_array_api.py` covering it are dropped)
- [x] Same class of leak for `add`/`subtract` with `uncertainty_correlation != 0`: the wrapper still delegates to astropy's `_propagate_add_sub`, whose correlation term uses `np.sqrt(this * other)` (`nduncertainty.py:742`), so a correlated add/subtract through the wrapper fails on strict with `TypeError: Expected Array or Python scalar; got numpy.ndarray`. Uncorrelated add/subtract is fine. Not covered by any test and not on the strict failure list; found while covering #993's multiply/divide override. Fix is the same shape as #993 (the mixin owns the formula with `xp.sqrt`) — **PR #997** (`test_wrapped_arithmetic_correlated_uncertainty` now covers `add`/`subtract` too)
- [x] `transform_image` applies `transform_func` to the bool mask directly (`core.py:1153`); numpy promotes `10 * mask` silently, strict raises. Cast the mask to a numeric dtype before transforming (the `mask > 0` afterwards already assumes a numeric result). 1 strict failure (`test_transform_image[True-True]`) — **PR #995** (cast to the data dtype; the test body also discarded the `xpx.at(...).set()` result, same class as #963, and now checks the transformed mask's contents)
- [x] #982 `combine()` does not normalise a raw module passed as `array_package`, so `combine(files, array_package=dask.array)` fails; `Combiner.__init__` was fixed in #976 and the two entry points should agree — **PR #997** (the crash itself went inert with #995, whose `device=`-free conversion a raw `dask.array` accepts, and `combine()` re-derives `xp` from the converted array anyway; the normalisation now matches `Combiner`, an array is still accepted as `array_package`, and regression tests cover raw `numpy`/`dask.array`)
- [x] `combine(dtype=...)`/`Combiner(dtype=...)` hand the user's dtype straight to `xp.astype`/`xp.asarray` (`combiner.py:1104`, `:191`); a builtin Python `int` is a valid NumPy dtype but array-api-strict rejects it (`AttributeError: type object 'int' has no attribute '_np_dtype'`). 1 strict failure (`test_combiner_result_dtype`, whose `dtype=int` case is the point of the test), reachable since #998 cleared the `.multiply()` failure in front of it — **PR #999** (`core._namespace_dtype` resolves the name with `numpy.dtype` and looks it up on the namespace; the namespace's own dtype objects pass through)
- [x] `subtract_overscan` calls `len(oscan)` on the overscan array (`core.py:698`, `xp.arange(len(oscan))`); array-api-strict has no `__len__`. Masked on the strict job because `test_subtract_overscan_model` is xfailed on #933 (with `oscan.shape[0]` in its place the test stops in `astropy.modeling.fitting._convert_input`, so the marker is right), but it is a one-line ccdproc numpy-ism — **PR #999**
- [x] ~~`Combiner.scaling` setter has an array-API bug exposed when test bodies stop using numpy `.mean()`~~ — diagnosed 2026-08-25: not a setter bug. `test_3d_combiner_with_scaling` fails inside the test helper `_make_mean_scaler` (`test_combiner.py:52`, `.mean()`/`np.ma.average`), and `test_combiner_with_scaling` fails earlier on `ccd_data.multiply(3)` in the test body (astropy NDData arithmetic). Both moved to section 2
- [x] `Combiner._get_nan_substituted_data` calls the numpy method `self._data_arr_mask.any()` (`combiner.py:517`) — fixed in #988, along with the further numpy-isms the fix exposed in the same functions (`.sum(axis=0)` mask counts, `len()` on arrays, list passed to `xp.reshape`, int→float promotion before `sqrt`)
- [x] `combine()` memory sizing uses `.nbytes` on data/uncertainty/mask/flags — fixed in #988 via `_array_size_in_bytes` (element count × `finfo`/`iinfo` bit width)
- [x] Builtin `bool` passed where `xp.bool` is required, in the mask paths of `Combiner` and `_CCDDataWrapperForArrayAPI` — fixed in #988 (wrapper mask setters plus the test sites building masks in the strict namespace)
- [x] Combiner: median combine and the uncertainty calculation still drop to plain numpy (see comments around `combiner.py` median/uncertainty paths; #351)

### 2. Test-suite hygiene
- [x] #969 `test_subtract_overscan` calls numpy-only `.copy()` — fixed in #990 (the two `median=True` parametrizations went green with #989)
- [x] #970 raw `np.zeros_like` in `test_ccdproc_logging.py` / `test_rebin.py` — fixed in #990 (`test_rebin_ccddata[True-True]` stays red on astropy's numpy-forcing `CCDData.mask` setter, #983's blocker)
- [x] Numpy `.sum()` / `.mean()` method calls in `test_combiner.py` test bodies — fixed in #990, along with ellipsis indexing, int/float promotion and `xp.asarray(list)`→`xp.stack` cases
- [x] `test__overscan_schange` calls `xp.allclose` — fixed in #990
- [x] Six `test_combiner.py` tests build scaled inputs with astropy `CCDData.multiply(...)` on strict data (`test_combiner_with_scaling`, `test_combiner_with_scaling_uncertainty` ×3, `test_combiner_result_dtype`, `test_combine_overwrite_output`); astropy's `np.result_type` in `_prepare_then_do_arithmetic` cannot take a strict dtype. Build the inputs in the namespace (`CCDData(ccd.data * 3, unit=...)`) instead — this is the NDData-arithmetic limitation already listed in #940 and is what `_CCDDataWrapperForArrayAPI` exists for. 6 strict failures — **PR #998** (the four jax `backend_xfail` markers for the same `.multiply()` calls are dropped, they XPASS; `test_combiner_with_scaling_uncertainty[average|sum]` clear, and the other four stop on the next cause: `test_combine_overwrite_output` on `combine(output_file=)` (section 3), `test_combiner_result_dtype` on `combine(dtype=int)` (section 1), `test_combiner_with_scaling` and `test_combiner_with_scaling_uncertainty[median_combine]` on #929)
- [x] Five `combine()` tests feed FITS filenames or numpy `CCDData` without `array_package=` and then compare the (numpy) result against strict arrays with `xp.all(xpx.isclose(...))` (`test_combine_average_fitsimages`, `test_combine_numpyndarray`, `test_combine_average_ccddata`, `test_combine_limitedmem_fitsimages`, `test_combine_limitedmem_scale_fitsimages`). Pass `array_package=xp` (or compare in numpy). 5 strict failures — **PR #998** (`array_package=xp` plus `Combiner(ccd_list, xp=xp)` for the reference; `test_combine_average_ccddata` converts its `CCDData` list by hand because `array_package` is ignored for `CCDData` input)
- [x] `test_combiner_image_file_collection_input` / `test_combine_image_file_collection_input` write a `device1` strict `CCDData` with `ccd.write()`; `astropy.io.fits` cannot coerce it. Write a numpy copy. 2 strict failures — **PR #998** (`numpy_copy`/`numpy_ccddata` helpers in `pytest_fixtures.py`; also clears `test_writeable_after_combine[average|sum_combine]`, while `[median_combine]` stays on #929)
- [x] `test_subtract_overscan_model` divides an integer strict array by a float (`test_ccdproc.py:304`); cast `np_mgrid` to float first. 2 strict failures — expect these to then land on #933 (`astropy.modeling`) — **PR #998** (they do, via the `len(oscan)` numpy-ism listed in section 1; now `backend_xfail("array-api-strict")` citing #933)
- [x] `_make_mean_scaler` (`test_combiner.py:52`) uses `.mean()` and `np.ma.average` — rewrite in `xp`. 2 strict failures (`test_3d_combiner_with_scaling`, and `test_combine_ccd_with_uncertainty_and_mask_from_fits[function]` once #995 lands) — **PR #998** (both pass)
- [x] #946 device-mismatch noise from inline array creation in strict-job test bodies — after the section-1 `device=` fixes above the remaining device errors are all astropy setters/`Quantity`, not test bodies; re-check once those land and close if nothing is left — re-checked after #998: none of the 15 remaining strict failures is a test-body device error; **#946 closed 2026-08-25**
- [x] `test_generator_ccds_without_unit` fails on the strict job because of an **astropy bug**, not test ordering per se: the `_arithmetic` decorator in `astropy/nddata/ccddata.py` sets the module-global `_config_ccd_requires_unit = False` and restores it with no `try`/`finally`, so *any* exception inside `CCDData.add/subtract/multiply/divide` leaves the unit requirement disabled process-wide. `test_unit_mismatch_behaves_as_expected` raises inside `.subtract()` by design, so the flag leaks on every backend — reproduced on plain numpy with just those two tests. The full numpy run stays green only because intervening tests (e.g. `test_gain`) complete a successful arithmetic pass, which resets the flag; on strict those tests fail too, so nothing repairs it. Filed upstream as astropy/astropy#20268; listed in #940 — **PR #998**: removing the `.multiply()` calls from `test_combiner.py` also removed the incidental successful arithmetic that had been resetting the flag, so the leak surfaced on plain numpy too; the test now resets `astropy.nddata.ccddata._config_ccd_requires_unit = True` itself and passes on every backend regardless of order, and the marker is gone
- [x] #943 jax `ccds()` platform-dependent `ValueError` behaviour (Linux CI only) — same astropy leak as above; the xfail reason at `test_image_collection.py:438` should cite astropy/astropy#20268 rather than "platform-dependent" — **PR #998** drops the marker altogether (see the previous bullet); **#943 closed 2026-08-25**
- [x] Correct the xfail reason added in #958 that cites astroscrappy (astroscrappy is monkeypatched out in that test) — already done on `main` in `d33d7cf`

### 3. Design issues / blocked on upstream (tracked in #940)
- [x] #929 `Combiner.sigma_clipping` and the default `sigma_func` densify via `astropy.stats` — **done: PR #1000 (`sigma_func`) and PR #1001 (`sigma_clipping`), both merged 2026-08-30**, numpy unchanged, every other namespace xp-native; leaves strict at 2 (#936, #983). History: 8 strict failures (5 via `median_absolute_deviation` at `core.py:1291`, 3 via `sigma_clip` at `combiner.py:491`); the largest remaining bucket once section 1 is done. After #998 it is **11**: the three `median_combine` callers `test_combiner_with_scaling`, `test_combiner_with_scaling_uncertainty[median_combine]` and `test_writeable_after_combine[median_combine]` reach the default `sigma_func` once their first cause is cleared. With #999 merged the split is **8 via `sigma_func`** (`core.py:1409`; `test_sigma_func_for_ccddata`, `test_combiner_dtype`, `test_combiner_median`, `test_combiner_with_scaling`, `test_combine_result_uncertainty_and_mask[median_combine-*]`, `test_writeable_after_combine[median_combine]`, `test_combiner_with_scaling_uncertainty[median_combine]`) and **3 via `sigma_clipping`** (`combiner.py:480`; `test_combiner_sigmaclip_{high,low,single_pix}`). The `sigma_func` half is cheap on the ccdproc side (MAD = `median(|x - median(x)|)` on top of the `_nanfuncs.py` median); the `sigma_clipping` half is harder because the method forwards `**kwd` and string/callable `func`/`dev_func` to astropy — options are an xp-native clip or `backend_xfail` markers
- [x] `combine(output_file=...)` writes the namespace result with `ccd.write(output_file, ...)` (`combiner.py:1273`); `astropy.io.fits` needs NumPy, so on strict it fails with `AttributeError: 'Array' object has no attribute 'astype'` (`ccddata.py:399`). 1 strict failure (`test_combine_overwrite_output`, reachable since #998). Same class as the CPU-only policy (#935): the writer needs a NumPy copy — **PR #999** (new `core._to_numpy` host-copy helper, device-generic; the writer gets a NumPy `CCDData` and the namespace result is returned unchanged. This is the first library-side host-copy helper, so #935 can build on it)
- [ ] #936 `Quantity`/units handling with non-numpy arrays — 1 strict failure (`test_unit_mismatch_behaves_as_expected`)
- [ ] #984 `cosmicray_median` does not exclude masked pixels from the median filter itself, so bright masked regions pull the local median (follow-up to #932/#979; runs in numpy, so it also belongs with the CPU-only policy in #935)
- [ ] #983 consider `marray` as a uniform masked-array representation across backends; bounded by astropy's numpy-only `NDData.mask` setter
- [ ] #933 `subtract_overscan` model fitting goes through numpy-only `astropy.modeling`
- [ ] #930 `wcs_project` mixes numpy output from `reproject` with xp operations
- [ ] #940 upstream blockers: `astropy.stats.sigma_clip`, `median_absolute_deviation`, `block_reduce`/`block_replicate`, NDData arithmetic (`np.result_type` in `_prepare_then_do_arithmetic`, `np.abs` in `_propagate_multiply_divide`; the `_CCDDataWrapperForArrayAPI` workaround), the `_config_ccd_requires_unit` leak (astropy/astropy#20268), `astropy.modeling`, `Quantity`, `reproject` (CPU-only), astroscrappy (permanent → document under the CPU-only policy, #935), `__array__(copy=)` warning (#934)

### 4. Tooling / infrastructure
- [ ] #945 escape-baseline keys are fragile across Python versions (PEP 709)
- [ ] #947 refactor `_escape_triage.py`; register pytest hooks explicitly
- [ ] #938 sparse backend as a densification detector (sub-items in that issue)
- [ ] Escape logger only patches `np.asarray`/`np.asanyarray`/`np.ma.asanyarray`; it misses `np.array()` and ufunc-driven coercion, so it is a sampler rather than a complete detector
- [ ] Decide between a parametrized `xp` fixture (SciPy-style, all backends in one run) and the current one-backend-per-`CCDPROC_ARRAY_LIBRARY` approach
- [x] Promote the `array-api-strict` job from soft to required — **done in #1005 (merged 2026-08-30)**: the last ccdproc-owned strict failure (`flat_correct` using the numpy-only `mask.any()` method) is fixed, #936/#983 carry `backend_xfail` markers, `py313-strict` moved into the regular `ci-tests` matrix, and the expected-failures job + `strict_status.yml` are deleted. (Marking the job *required* in branch protection is a separate admin step; `main` currently has no required status checks configured at all.) History: sections 1–2 have landed (2026-08-26); the residue is exactly 13 upstream-blocked tests (11 × #929, #936, #983), of which 11 are fixed by #1000/#1001; the residue after those is #936 and #983, which need `backend_xfail` markers citing the astropy items so the job can be green
- [x] #1004 hoist `None`/tuple axis handling into `_nanfuncs._setup` — **done in #1006 (merged 2026-08-31)**: `_setup` now takes an int, `None` or a tuple/list of axes and returns a `restore` callable mapping full-shape arrays back to the caller's layout, so every `_nanfuncs` reduction gained tuple/`None` axis support for free, `core._mad_fallback` shrank to the median tier plus a delegation to `nanmad`, and `Combiner.sigma_clipping` off numpy accepts the same axis forms as `astropy.stats.sigma_clip` (int, `None`, tuple or list). Review caught two edge cases before merge: `np.bool_` axes were silently treated as axis 1 on numpy 2.0 (now `TypeError`), and the tuple merge reshaped with `-1`, which raises when a kept axis has size 0 (`numpy.median` itself has this bug; the fallbacks now return an empty result like numpy's other reductions). Behaviour note: a bool or otherwise non-integer `axis` now raises `TypeError` rather than `NotImplementedError`
- [ ] Decide whether CuPy and/or torch get a CI job

### 5. Close out
- [x] Re-triage the `backend_xfail` markers after the fixes above and drop the stale ones — **done 2026-08-30 (#1005 dropped the stale `test_flat_correct_masked_flat_with_immutable_array` marker; the 30 remaining markers are all genuine numpy-only boundaries)**. History: the five known-stale ones below are narrowed in **PR #996** (merged; strict XPASS 5 → 0), and #998 drops five jax markers (four `.multiply()` ones in `test_combiner.py`, the `test_generator_ccds_without_unit` one). Known stale before #996: `test_cosmicray.py::test_cosmicray_median_masked_column[masked_array-1.0]` and `[masked_array-None]` XPASS on the strict job, and their reason (a `scipy.ndimage.median_filter` device failure) no longer applies after #979; the `test_cosmicray_gain_correct_uncertainty_namespace` marker (`test_cosmicray.py:154`) XPASSes for the three `gain_apply=False` cases and should be narrowed to the `gain_apply=True` parametrizations
- [ ] Re-run the suite against CuPy and close the parts of #909–#913 that are subsumed by the issues above
- [ ] Confirm `docs/array_api.rst` and `CHANGES.rst` reflect the final set of supported backends

Contributor guide

Open the contributing guide

Research direction

Start by reading the migration checklist in this issue, then inspect the named entry points in ccdproc/core.py, ccdproc/combiner.py, and _ccddata_wrapper_for_array_api.py. Run the py313-strict and backend test jobs to identify an unclaimed remaining item. Done means the selected migration work is addressed without new failures and the strict array-API checks remain clean.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, testing-qa
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.