AI-Hypercomputer / AI-Hypercomputer/google-cloud-mldiagnostics

machinelearning_run() Fatal Python error: Aborted — libtpu re-import causes protobuf double-registration (1.0.2 wheel)

Offen
#3 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
14
Forks
4
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Summary
`machinelearning_run()` **hard-crashes the process** (`Fatal Python error: Aborted`) when `libtpu` is already loaded by JAX (i.e. any real JAX-on-TPU workload). `libtpu_metric._initialize()` does `from libtpu import sdk`, whose C-extension re-registers protobuf descriptors that are already in the global pool → `descriptor.cc` `Check failed`. `_initialize()` only catches `ImportError`, which cannot catch a C++ `abort()`.

## Environment
- `google-cloud-mldiagnostics` **1.0.2** (the version shipping in current TPU images)
- GKE TPU (tpu7x), plain-SPMD SFT (local libtpu present), JAX 0.10.0

## Error (verbatim)
```
E0000 descriptor_database.cc:633 File already exists in database: google/protobuf/timestamp.proto
F0000 descriptor.cc:2236 Check failed ...
Fatal Python error: Aborted
```
Call path: `machinelearning_run` → `get_software_config` → `get_libtpu_version` → `libtpu_metric._initialize` → `from libtpu import sdk`. Reproduced deterministically — all 16 workers crashlooped (RESTART climbing 0→1→2).

## Root cause
The SDK re-imports the `libtpu` C-extension instead of reusing the module JAX already loaded; the second registration of `timestamp.proto` is fatal.

## Note on HEAD
HEAD commit `08d554d` ("Enforce JAX-style RTLD_LOCAL and RTLD_DEEPBIND dlopen isolation during LibTPU SDK import") adds `RTLD_LOCAL|RTLD_DEEPBIND` and broadens the except — which likely mitigates this. **But it is not in the released 1.0.2/1.0.3 wheel** that ships in the images, so deployed workloads still abort.

## Asks
1. Cut a release containing `08d554d` so deployed images stop aborting.
2. Consider, in addition to dlopen isolation: if `libtpu` is already in `sys.modules`, reuse it / read the version via `importlib.metadata.version("libtpu")` without importing the C-extension; and broaden the guard to `except BaseException` so a probe failure degrades to `"n/a"` rather than aborting the trainer.

— Reported via Navi on behalf of @lokic233 (Meta MRS-CE).

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.