ashvardanian / ashvardanian/NumKong

Feature: Batched 2-D and 3-D resampling for resize, warp, and remap

Open
#362 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
1.9k
Forks
130
Avg merge
18h 28m
Merged PRs (30d)
3

Description

## Workload

Albucore calls `cv2.resize`, `warpAffine`, `warpPerspective`, and `remap` for one image at a time. Videos, image batches, and volumes require Python loops or reshaping around APIs that assume one 2D image.

These operations share a lower-level task: sample a channel-last Tensor at transformed or explicitly provided coordinates.

This has become a concrete foundation for the AlbumentationsX 3D roadmap. One true 3D sampler would unblock [`Affine3D`](https://github.com/albumentations-team/AlbumentationsX/issues/328), [`ElasticTransform3D`](https://github.com/albumentations-team/AlbumentationsX/issues/327), [`Anisotropy3D`](https://github.com/albumentations-team/AlbumentationsX/issues/331), shape-based `Resize3D`, and later spacing-aware resampling.

## Requested capability

A lower-level resampling API could support the family:

```python
out = nk.remap(
src,
coordinates,
interpolation="linear",
border_mode="constant",
border_value=0,
out=None,
)
```

Convenience `resize` and matrix-based `warp` functions can build on the same kernels without requiring callers to materialize a dense coordinate grid when the mapping is affine.

## Phase 1: batched 2D

- Input: `(..., H, W, C)` with any number of leading batch dimensions.
- Coordinates: shared `(H2, W2, 2)` map or batch-broadcastable maps.
- Output: `(..., H2, W2, C)`.
- Interpolation: nearest and linear, including explicitly defined exact variants.
- Border modes: constant, replicate, reflect, and wrap.
- Dtypes: `uint8` and `float32`.
- Shared or per-batch affine/perspective matrices.
- Treat channels as an arbitrary trailing dimension; include C=1, C=9, and C=129 in correctness tests.

## Phase 2: true 3D volumes

- Input: `(..., D, H, W, C)` and output `(..., D2, H2, W2, C)`.
- Coordinates: shared `(D2, H2, W2, 3)` map or batch-broadcastable maps.
- Resize by explicit `(D2, H2, W2)` output shape or scale per spatial axis.
- Shared or per-volume 3x4/4x4 affine transforms.
- Nearest-neighbor and trilinear interpolation first. Nearest is required for masks and labels; trilinear is required for intensity volumes.
- Constant, replicate, reflect, and wrap borders.
- `uint8` and `float32`, arbitrary channel counts, non-cubic shapes, `D=1`, and singleton spatial axes.
- Do not treat depth as a batch: rotations and dense maps must be able to mix or displace all three spatial axes.

## Semantics to define

- Pixel-center convention and coordinate order.
- Forward versus inverse transform matrices.
- Rounding for nearest-neighbor sampling.
- Integer interpolation and saturation.
- Border behavior for coordinates exactly on and just outside the boundary.
- Aliasing rules for `out=`.
- Bit-exact contracts for nearest-exact and linear-exact interpolation.

## Extensions

- Cubic and Lanczos interpolation for every channel count, including C>4.
- Area interpolation and antialiasing for downscaling.
- Precomputed coefficient tables for repeated resize shapes.

## Baselines and tests

Use OpenCV as the 2D compatibility baseline and SciPy/PyTorch as independent 3D references after explicitly aligning coordinate conventions. Test identity maps, all 90-degree rotations, half-pixel translations, degenerate dimensions, negative coordinates, large batches, grayscale C=1, C=9, and C=129.

Benchmark single HWC calls, batched NHWC calls, and true DHWC/NDHWC volume calls. Include `(32, 256, 256, C)` and `(64, 128, 128, C)`, runtime, and peak memory. The batch case should include the current Python loop around OpenCV; the volume case should use a semantically equivalent 3D reference rather than a slice-wise 2D loop.

Related tracker: #313. Related OpenCV batch-only requests: [opencv/opencv#29588](https://github.com/opencv/opencv/issues/29588) and [opencv/opencv#29590](https://github.com/opencv/opencv/issues/29590).

Contributor guide

Open the contributing guide

Research direction

Start at the proposed nk.remap entry point and define the coordinate, transform, interpolation, border, dtype, and aliasing semantics against the OpenCV 2D baseline. Use SciPy and PyTorch as independent 3D references after aligning conventions, then cover the listed identity, rotation, degenerate-axis, negative-coordinate, channel-count, batch, and volume cases. Done means both phases and the specified benchmark comparisons are tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, opencv, python, pytorch
Domain
computer-vision, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.