JuliaDataCubes / JuliaDataCubes/YAXArrays.jl

Can we get YAXArrays to automatically parse datetime variables

Open
#573 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
132
Forks
25
PR merge metrics
No merged PRs in 30d

Description

It seems that YAXArrays does a great job at parsing DateTime dimensions but not DateTime variables. Here's an example that points to open data.

load packages and data

using YAXArrays
using Zarr
path = "https://its-live-data.s3.amazonaws.com/datacubes/v2-updated-october2024/S70E070/ITS_LIVE_vel_EPSG3031_G0120_X1250000_Y350000.zarr";
store = zopen(path); 
ds = YAXArrays.open_dataset(store); 

inspect mid_date dimension

ds.mid_date
julia> mid_date [DateTime("2007-07-21T02:40:09.758"), …, DateTime("2024-12-19T02:19:33.907")]

inspect acquisition_date_img1 variable

ds.acquisition_date_img1
┌ 22242-element YAXArray{Float64, 1} ┐
├────────────────────────────────────┴─────────────────────────────────────────────────────────────────── dims ┐
  ↓ mid_date Sampled{DateTime} [DateTime("2007-07-21T02:40:09.758"), …, DateTime("2024-12-19T02:19:33.907")] Unordered Irregular Points
├──────────────────────────────────────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any} with 6 entries:
  "units"         => "days since 1970-01-01"
  "calendar"      => "proleptic_gregorian"
  "name"          => "acquisition_date_img1"
  "standard_name" => "image1_acquition_date"
  "description"   => "acquisition date and time of image 1"
  "_FillValue"    => NaN
├─────────────────────────────────────────────────────────────────────────────────────────────── loaded lazily ┤
  data size: 173.77 KB

Notice that acquisition_date_img1 has not been converted to DateTime

ds.acquisition_date_img1[1]
julia> 13499.111201768137

Is there a way that we could get YAXArrays to automatically parse DateTime variables?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the YAXArrays.open_dataset entry point and trace how CF metadata such as units and calendar is applied to dimensions versus variables. Reproduce the provided Zarr example and add a regression test for acquisition_date_img1 using its days-since-epoch metadata. Done means eligible datetime variables are exposed as DateTime values without breaking numeric variables.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.