tensorflow / tensorflow/tensorboard

Diagnose and fix SIGSEGV/SIGABRTs

Open
#2,987 2 comments 0 reactions 1 assignee View on GitHub

@davidsoergel is already working on this.

Since Dec 2, 2019.

type:build/install
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

All Travis builds are currently failing in the example plugin smoke test:

+ python -m tensorboard_plugin_example.demo
2019-12-02 21:26:34.264757: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2019-12-02 21:26:34.264791: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2019-12-02 21:26:34.264821: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (travis-job-48f6ca02-7a68-439f-8e39-c272ce59d7d3): /proc/driver/nvidia/version does not exist
2019-12-02 21:26:34.265298: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-02 21:26:34.275750: I tensorflow/core/platform/profile_utils/cpu_utils.cc:101] CPU Frequency: 2300000000 Hz
2019-12-02 21:26:34.275976: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4bac2a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-12-02 21:26:34.275990: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
*** Error in `python': corrupted size vs. prev_size: 0x0000000001805c60 ***
Aborted (core dumped)

I reported this as an upstream issue two weeks ago, but it hasn’t been
fixed yet: https://github.com/tensorflow/tensorflow/issues/34427. At
that time, we were also in the process of migrating from Travis .com to
.org. That migration “happened” to fix the problem, but that may have
been a caching issue. We’ve pinned ourselves to non-broken TF nightlies
(#2947), but that doesn’t appear to have sufficed.

As a quick fire extinguisher, I’m disabling the test, but this is
actually a pretty important test to keep around: it’s our sole
integration test for the dynamic plugin system.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.