AI-Hypercomputer / AI-Hypercomputer/google-cloud-mldiagnostics
machinelearning_run() Fatal Python error: Aborted — libtpu re-import causes protobuf double-registration (1.0.2 wheel)
- Vorherrschende Sprache
- Python
- Sterne
- 14
- Forks
- 4
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
## Summary
`machinelearning_run()` **hard-crashes the process** (`Fatal Python error: Aborted`) when `libtpu` is already loaded by JAX (i.e. any real JAX-on-TPU workload). `libtpu_metric._initialize()` does `from libtpu import sdk`, whose C-extension re-registers protobuf descriptors that are already in the global pool → `descriptor.cc` `Check failed`. `_initialize()` only catches `ImportError`, which cannot catch a C++ `abort()`.
## Environment
- `google-cloud-mldiagnostics` **1.0.2** (the version shipping in current TPU images)
- GKE TPU (tpu7x), plain-SPMD SFT (local libtpu present), JAX 0.10.0
## Error (verbatim)
```
E0000 descriptor_database.cc:633 File already exists in database: google/protobuf/timestamp.proto
F0000 descriptor.cc:2236 Check failed ...
Fatal Python error: Aborted
```
Call path: `machinelearning_run` → `get_software_config` → `get_libtpu_version` → `libtpu_metric._initialize` → `from libtpu import sdk`. Reproduced deterministically — all 16 workers crashlooped (RESTART climbing 0→1→2).
## Root cause
The SDK re-imports the `libtpu` C-extension instead of reusing the module JAX already loaded; the second registration of `timestamp.proto` is fatal.
## Note on HEAD
HEAD commit `08d554d` ("Enforce JAX-style RTLD_LOCAL and RTLD_DEEPBIND dlopen isolation during LibTPU SDK import") adds `RTLD_LOCAL|RTLD_DEEPBIND` and broadens the except — which likely mitigates this. **But it is not in the released 1.0.2/1.0.3 wheel** that ships in the images, so deployed workloads still abort.
## Asks
1. Cut a release containing `08d554d` so deployed images stop aborting.
2. Consider, in addition to dlopen isolation: if `libtpu` is already in `sys.modules`, reuse it / read the version via `importlib.metadata.version("libtpu")` without importing the C-extension; and broaden the guard to `except BaseException` so a probe failure degrades to `"n/a"` rather than aborting the trainer.
— Reported via Navi on behalf of @lokic233 (Meta MRS-CE).
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.