AI-Hypercomputer / AI-Hypercomputer/google-cloud-mldiagnostics

machinelearning_run() Fatal Python error: Aborted — libtpu re-import causes protobuf double-registration (1.0.2 wheel)

Abierto
#3 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
14
Forks
4
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Summary
`machinelearning_run()` **hard-crashes the process** (`Fatal Python error: Aborted`) when `libtpu` is already loaded by JAX (i.e. any real JAX-on-TPU workload). `libtpu_metric._initialize()` does `from libtpu import sdk`, whose C-extension re-registers protobuf descriptors that are already in the global pool → `descriptor.cc` `Check failed`. `_initialize()` only catches `ImportError`, which cannot catch a C++ `abort()`.

## Environment
- `google-cloud-mldiagnostics` **1.0.2** (the version shipping in current TPU images)
- GKE TPU (tpu7x), plain-SPMD SFT (local libtpu present), JAX 0.10.0

## Error (verbatim)
```
E0000 descriptor_database.cc:633 File already exists in database: google/protobuf/timestamp.proto
F0000 descriptor.cc:2236 Check failed ...
Fatal Python error: Aborted
```
Call path: `machinelearning_run` → `get_software_config` → `get_libtpu_version` → `libtpu_metric._initialize` → `from libtpu import sdk`. Reproduced deterministically — all 16 workers crashlooped (RESTART climbing 0→1→2).

## Root cause
The SDK re-imports the `libtpu` C-extension instead of reusing the module JAX already loaded; the second registration of `timestamp.proto` is fatal.

## Note on HEAD
HEAD commit `08d554d` ("Enforce JAX-style RTLD_LOCAL and RTLD_DEEPBIND dlopen isolation during LibTPU SDK import") adds `RTLD_LOCAL|RTLD_DEEPBIND` and broadens the except — which likely mitigates this. **But it is not in the released 1.0.2/1.0.3 wheel** that ships in the images, so deployed workloads still abort.

## Asks
1. Cut a release containing `08d554d` so deployed images stop aborting.
2. Consider, in addition to dlopen isolation: if `libtpu` is already in `sys.modules`, reuse it / read the version via `importlib.metadata.version("libtpu")` without importing the C-extension; and broaden the guard to `except BaseException` so a probe failure degrades to `"n/a"` rather than aborting the trainer.

— Reported via Navi on behalf of @lokic233 (Meta MRS-CE).

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.