developmentseed / developmentseed/async-tiff

Uint sample data with a non-byte-aligned bit depth (e.g. NBITS=15) fails decoding instead of unpacking

Open
#334 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
120
Forks
15
Avg merge
1d 17h
Merged PRs (30d)
9

Description

## Summary

For `SampleFormat::Uint` samples with a bit depth that isn't 1/8/16/32/64 (e.g. `BitsPerSample = 15`, a GDAL `NBITS=15`-style bit-packed encoding), `DataType::from_tags` falls through to `None`:

https://github.com/developmentseed/async-tiff/blob/056aad86fc8993fcf401fe2a5d2e84b92dc25ca5/src/data_type.rs#L75-L89

That `None` then reaches `Array::try_new`, whose only length check compares the *raw decompressed byte count* against a pixel-count-based `expected_len` that assumes 1 byte per element for the `None` case:

https://github.com/developmentseed/async-tiff/blob/056aad86fc8993fcf401fe2a5d2e84b92dc25ca5/src/array.rs#L45-L53

For a real Sentinel-2 asset with `BitsPerSample=15`, decompression correctly produces the tightly-packed byte buffer (`512 * 512 * 15 bits / 8 = 491520` bytes for a 512×512 tile), but the check expects `512 * 512 * 1 = 262144` — so every tile fails with a generic, confusing error rather than a clear "unsupported bit depth" message (or, ideally, correct unpacking).

```
General error: Internal error: incorrect shape or data length passed to Array::try_new. Got data length 491520, expected 262144
```

## Reproduction

Planetary Computer serves at least some Sentinel-2 L2A COGs with `BitsPerSample=15` (presumably a GDAL `NBITS=15` creation option, since Sentinel-2 L2A reflectance digital numbers never exceed 15 bits' worth of range). Every tile of every band asset I tested for MGRS grid squares `T23LLJ`/`T23LLH` in that collection hits this — not an isolated corrupt file.

STAC item: `S2A_MSIL2A_20240905T132231_N0511_R038_T23LLJ_20240905T183000` in Planetary Computer's `sentinel-2-l2a` collection (`https://planetarycomputer.microsoft.com/api/stac/v1/collections/sentinel-2-l2a/items/S2A_MSIL2A_20240905T132231_N0511_R038_T23LLJ_20240905T183000`).

`B08` asset (unsigned; Planetary Computer's blob storage requires a SAS token — sign via `https://planetarycomputer.microsoft.com/api/sas/v1/sign?href=` and use the returned `href`, which is valid for a few hours):

```
https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/23/L/LJ/2024/09/05/S2A_MSIL2A_20240905T132231_N0511_R038_T23LLJ_20240905T183000.SAFE/GRANULE/L2A_T23LLJ_A048083_20240905T132232/IMG_DATA/R10m/T23LLJ_20240905T132231_B08_10m.tif
```

Its IFD:

```
image_width: 10980, image_height: 10980
tile_width: 512, tile_height: 512
samples_per_pixel: 1
bits_per_sample: [15]
sample_format: [Uint]
compression: Deflate
predictor: None
```

Decoding tile `(0, 0)` — a full, non-edge, non-cropped tile, so this isn't the edge-tile/predictor-width issue from #173 — fails immediately with the error above. I confirmed the same failure across several other tile coordinates (corners, edges, center) too.

## This isn't a corrupted source file

Brazil Data Cube (`https://data.inpe.br/bdc/stac/v1`, collection `S2_L2A-1`) independently re-encodes the *same underlying ESA Sentinel-2 L2A archive* through its own COG pipeline, and happens to carry the exact same STAC item id for the exact same physical acquisition. Its `B08` asset for the identical scene (publicly readable, no signing needed):

```
https://data.inpe.br/bdc/data/S2_L2A/v001/23/L/LJ/2024/09/S2A_B08_20240905T132231_N0511_R038_T23LLJ_20240905T183000.tif
```

Its IFD — identical in every respect except the bit depth:

```
image_width: 10980, image_height: 10980
tile_width: 512, tile_height: 512
samples_per_pixel: 1
bits_per_sample: [16]
sample_format: [Uint]
compression: Deflate
predictor: None
```

...and this one decodes tile `(0, 0)` fine (`decoded_len=262144`, matching the expected shape exactly). Same scene, same tiling, same compression, same predictor — the only difference is the bit-packed 15-bit encoding, which is what actually triggers the failure.

## Suggested fix direction

`DataType::from_tags` currently only recognizes byte-aligned bit depths for `Uint`/`Int` (plus the existing 1-bit `Bool` special case in `Array::try_new`). Generalizing that special case — MSB-first bit unpacking for arbitrary `N` between 1 and 32, with each tile *row* padded to a byte boundary per the TIFF6 convention — into an `N`-bit integer path (not just `Bool`) would fix this class of bug outright, for any bit depth a GDAL `NBITS` creation option might produce, not just 15.

Failing that, at minimum `Array::try_new`/`from_tags` should distinguish "genuinely unsupported bit depth" from "shape mismatch" and surface a clearer error (e.g. `TiffUnsupportedError::UnsupportedBitDepth(u16)`) instead of the current message, which reads like an internal invariant violation rather than a missing feature.

Happy to follow up with a PR implementing the general bit-unpacking path if that direction sounds right to the maintainers.

---

*This issue was drafted with the assistance of [Claude Code](https://claude.com/claude-code), based on real debugging against the live Planetary Computer and Brazil Data Cube STAC APIs.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading src/data_type.rs around DataType::from_tags and src/array.rs around Array::try_new, including the existing 1-bit Bool handling. Reproduce the failure with the described 512×512, BitsPerSample=15 tile and trace the length check. Done means either non-byte-aligned unsigned samples are unpacked correctly or unsupported depths produce a specific error rather than a shape-mismatch error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.