google / google/xarray-beam

Some notes on alignment with pangeo-forge-recipes around "keys"

Aperta
#32 1 commento 4 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
170
Fork
15
Merge medio
18h 27m
PR unite (30g)
1

Descrizione

At today's Pangeo Forge meeting, @alxmrs told us a bit more about Xarray Beam. I'm happy to note that we are aligning a bit around certain abstractions. I'm not proposing to merge or share code or anything at this stage. Just happy to see that we have independently arrived at some similar concepts. Comparing side by side may also suggest improvements.

### Xarray Beam

https://xarray-beam.readthedocs.io/en/latest/data-model.html#keys-in-xarray-beam

> To keep track of how individual records could be combined into a larger (virtual) dataset, Xarray-Beam defines a Key object. > Key objects consist of:
> - offsets: integer offests for chunks from the origin in an immutabledict
> - vars: The subset of variables included in each chunk, either as a frozenset, or as None to indicate “all variables”.

### Pangeo Forge

We currently call this an "index," although I agree that key is a better name. Our use of indexes is less well documented as it is not really public API but more of an internal thing. We mention it here: https://pangeo-forge.readthedocs.io/en/latest/recipe_user_guide/file_patterns.html#inspect-a-filepattern

> The index is its own special type of object used internally by recipes, a `pangeo_forge_recipes.patterns.Index`, (which is basically a tuple of one or more `pangeo_forge_recipes.patterns.DimIndex` objects).

The code is pretty straightforward and the comments explain things better.

https://github.com/pangeo-forge/pangeo-forge-recipes/blob/49997cb52cff466bd394c1348ef23981e782a4d9/pangeo_forge_recipes/patterns.py#L70-L114

These indexes are used both by FilePatterns (see #31) as well as for parallel writing of Zarr datasets.

cc @cisaacstern

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia leggendo la documentazione di Xarray Beam keys e le righe 70-114 referenziate di pangeo_forge_recipes/patterns.py. Confronta queste astrazioni con la gestione degli indici in FilePatterns e nella scrittura parallela di Zarr. L'issue non definisce una modifica concreta né criteri di accettazione, quindi prima dell'implementazione è necessaria una decisione dei maintainer.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
data-engineering, distributed-systems
Tipo di issue
Refactoring
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.