JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

Persisting arrays on disk using zarr backend does not encode data using scale_factor/offset

Open
#495 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

Hello, I am processing data stored in zarr format. Most of it are stored on disk encoded following CF conventions. For instance I have a YAXArray

┌ 1500×1200 YAXArray{Union{Missing, Float64}, 2} ┐
├────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────── dims ┐
  ↓ columns Sampled{Int64} 1:1500 ForwardOrdered Regular Points,
  → rows    Sampled{Int64} 1:1200 ForwardOrdered Regular Points
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any} with 10 entries:
  "units"         => "K"
  "name"          => "s7_bt_in"
  "coordinates"   => "latitude longitude x y"
  "short_name"    => "s7_bt_in"
  "add_offset"    => 283.73
  "long_name"     => "gridded pixel brightness temperature for channel s7 (1km TIR grid, nadir view)"
  "missing_value" => -32768
  "scale_factor"  => 0.01
  "standard_name" => "toa_brightness_temperature"
  "_FillValue"    => -32768
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── loaded lazily ┤
  data size: 13.73 MB
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

The underlying data is a CFDiskArray highlighting that the original zarray is Int16

1500×1200 DiskArrayTools.CFDiskArray{Union{Missing, Float64}, 2, Int16, ZArray{Int16, 2, Zarr.BloscCompressor, Zarr.ConsolidatedStore{DirectoryStore}}, Float64}

Chunked: (
    [1500]
    [1200]
)

When I save the data on disk using savecube or savedataset the array is stored as float64, and not encoded/packed in Int16.

For information, the original YAXArray is created with open_dataset from an existing zarr file and correctly decode/unpack the data. At last, surprisingly (or not), when I read again the persisted file (stored in float64), the scale_factor is not applied twice.

Am I missing any options there ? since DiskArrayTools seems to implement such mechanisms at reading/writing.

Many thanks in advance for your feedbacks!
Vincent

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the persistence path using the YAXArray opened with open_dataset, then compare savecube and savedataset output with the original Zarr data. Inspect how DiskArrayTools handles scale_factor, add_offset, and the Int16 backing type; done means saved data preserves the expected encoded representation and does not apply scaling twice when reopened.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.