google-deepmind / google-deepmind/weathernext

GenCast autoregressive rollout fails with JaxArrayWrapper / DynamicJaxprTracer __array_ufunc__ TypeError on both GPU and Colab TPU

Open
#203 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.7k
Forks
986
PR merge metrics
No merged PRs in 30d

Description

Hi, would really appreciate some help with the following issue of getting a TypeError while running the GenCast autoregressive rollout. It fails with a TypeError related to xarray_jax.JaxArrayWrapper interacting with a DynamicJaxprTracer.

The failure occurs during the autoregressive rollout step `(chunked_prediction_generator_multiple_runs)` in the GenCast demo.

I can reproduce the same error on a local multi-GPU machine, and on Google Colab TPU (v5e-1 TPU with 2025.07 runtime version), and I think the error is related to the GenCast rollout path interacting with xarray_jax.

The simpler GraphCast forward pass works correctly.

Execution fails with:

```
TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(
, '__call__',
JitTracer(float32[]),
xarray_jax.JaxArrayWrapper(JitTracer(float32[1,1,181,360]))
): 'DynamicJaxprTracer', 'JaxArrayWrapper'
```

For reproduction:
Using GenCast demo notebook, the failure occurs in the Autoregressive rollout cell:
```
chunks = []
for chunk in rollout.chunked_prediction_generator_multiple_runs(
predictor_fn=run_forward_pmap,
rngs=rngs,
inputs=eval_inputs,
targets_template=eval_targets * np.nan,
forcings=eval_forcings,
num_steps_per_chunk=1,
num_samples=num_ensemble_members,
pmap_devices=jax.local_devices(),
):
chunks.append(chunk)

predictions = xarray.combine_by_coords(chunks)
```
Setup:

Machine:

Linux cluster

2× NVIDIA GPUs
Python environment:
```
Python 3.11
jax 0.4.38
jaxlib 0.4.38
numpy 2.4.2
xarray 2024.11.0
pandas 2.2.3
```
Devices detected:
```
devices: [CudaDevice(id=0), CudaDevice(id=1)]
backend: gpu
```
Error traceback ends with:
```
TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(
, '__call__',
JitTracer(float32[]),
xarray_jax.JaxArrayWrapper(JitTracer(float32[1,1,181,360]))
)
```

Contributor guide

Open the contributing guide

Research direction

Start with the GenCast demo notebook's autoregressive rollout cell and reproduce the failure using chunked_prediction_generator_multiple_runs on the listed GPU or TPU setup. Compare it with the working GraphCast forward pass and trace the interaction between the rollout inputs and xarray_jax wrappers. Done means the GenCast autoregressive rollout completes without the DynamicJaxprTracer __array_ufunc__ TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.