tensorflow / tensorflow/tensorboard
Diagnose and fix SIGSEGV/SIGABRTs
@davidsoergel is already working on this.
Since Dec 2, 2019.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
All Travis builds are currently failing in the example plugin smoke test:
+ python -m tensorboard_plugin_example.demo
2019-12-02 21:26:34.264757: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2019-12-02 21:26:34.264791: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2019-12-02 21:26:34.264821: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (travis-job-48f6ca02-7a68-439f-8e39-c272ce59d7d3): /proc/driver/nvidia/version does not exist
2019-12-02 21:26:34.265298: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-02 21:26:34.275750: I tensorflow/core/platform/profile_utils/cpu_utils.cc:101] CPU Frequency: 2300000000 Hz
2019-12-02 21:26:34.275976: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4bac2a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-12-02 21:26:34.275990: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
*** Error in `python': corrupted size vs. prev_size: 0x0000000001805c60 ***
Aborted (core dumped)
I reported this as an upstream issue two weeks ago, but it hasn’t been
fixed yet: https://github.com/tensorflow/tensorflow/issues/34427. At
that time, we were also in the process of migrating from Travis .com to
.org. That migration “happened” to fix the problem, but that may have
been a caching issue. We’ve pinned ourselves to non-broken TF nightlies
(#2947), but that doesn’t appear to have sufficed.
As a quick fire extinguisher, I’m disabling the test, but this is
actually a pretty important test to keep around: it’s our sole
integration test for the dynamic plugin system.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.