tensorflow / tensorflow/tensorflow

tf.raw_ops.WriteScalarSummary can segfault on GPU instead of raising a Python exception

Open
#113,060 3 comments 0 reactions 1 assignee View on GitHub

@Venkat6871 is already working on this.

Since Mar 24, 2026.

2.21.0 awaiting PR merge comp:ops type:bug
Dominant language
C++
Stars
200k
Forks
76.9k
Avg merge
2d 3h
Merged PRs (30d)
433

Description

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

source

TensorFlow version

2.21.0

Custom code

Yes

OS platform and distribution

Ubuntu 24.04.3 LTS x86_64

Mobile device

No response

Python version

3.10.20

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

13.0

GPU model and memory

No response

Current behavior?
System information
  • TensorFlow version: 2.21.0
  • Python version: 3.10.20
  • OS: Linux 6.17.0-19-generic x86_64
  • GPU: 4 x NVIDIA GeForce RTX 2080 Ti
  • Reproduced in: conda env tf_test
Description

I found a case where tf.raw_ops.WriteScalarSummary crashes the Python process on GPU instead of raising a normal Python exception.

In batch validation this showed up as a generic signal_kill, but when I reran the same script standalone, it consistently reproduced as:

  • Segmentation fault (core dumped)

This looks like a hard crash rather than expected input validation behavior.

Reproducer
import tensorflow as tf

def test_write_scalar_summary():
    logdir = "logs/"
    writer = tf.summary.create_file_writer(logdir)

    step = tf.constant(1, dtype=tf.int64)
    tag = tf.constant("test_tag", dtype=tf.string)

    values = [
        tf.constant(0.0, dtype=tf.float32),
        tf.constant(0.0, dtype=tf.float64),
        tf.constant(0, dtype=tf.int32),
        tf.constant(0, dtype=tf.int64),
        tf.constant(0, dtype=tf.float16),
        tf.constant([], dtype=tf.float32),
        tf.constant([[0]], dtype=tf.float32),
        tf.constant([[0, 0]], dtype=tf.float32),
    ]

    with writer.as_default():
        for value in values:
            try:
                tf.raw_ops.WriteScalarSummary(
                    writer=writer._resource,
                    step=step,
                    tag=tag,
                    value=value,
                )
                print(f"Successfully wrote summary for value: {value.numpy()}")
            except Exception as e:
                print(f"Failed to write summary for value: {value.numpy()} with error: {e}")

    writer.close()

with tf.device('/GPU:0'):
    test_write_scalar_summary()


### Standalone code to reproduce the issue

```shell
conda run --no-capture-output -n tf_test python reproduce.py
Actual behavior
The process terminates with a segfault:

Segmentation fault (core dumped)
Before the crash, TensorFlow initializes GPU devices normally.

Expected behavior
tf.raw_ops.WriteScalarSummary should reject unsupported inputs by raising a normal Python / TensorFlow exception such as InvalidArgumentError or ValueError, rather than crashing the whole process.

pip list
Package                  Version
------------------------ ----------
absl-py                  2.4.0
aiohappyeyeballs         2.6.1
aiohttp                  3.13.3
aiosignal                1.4.0
annotated-types          0.7.0
anyio                    4.12.1
astunparse               1.6.3
async-timeout            5.0.1
attrs                    25.4.0
cachetools               7.0.5
certifi                  2026.2.25
charset-normalizer       3.4.5
colorlog                 6.10.1
distro                   1.9.0
exceptiongroup           1.3.1
flatbuffers              25.12.19
frozenlist               1.8.0
gast                     0.7.0
google-pasta             0.2.0
grpcio                   1.78.0
h11                      0.16.0
h5py                     3.14.0
httpcore                 1.0.9
httpx                    0.28.1
idna                     3.11
jiter                    0.13.0
keras                    3.12.1
libclang                 18.1.1
markdown-it-py           4.0.0
mdurl                    0.1.2
ml_dtypes                0.5.4
multidict                6.7.1
namex                    0.1.0
networkx                 3.4.2
numpy                    2.2.6
nvidia-cublas-cu12       12.9.1.4
nvidia-cuda-cupti-cu12   12.9.79
nvidia-cuda-nvcc-cu12    12.9.86
nvidia-cuda-nvrtc-cu12   12.9.86
nvidia-cuda-runtime-cu12 12.9.79
nvidia-cudnn-cu12        9.20.0.48
nvidia-cufft-cu12        11.4.1.4
nvidia-curand-cu12       10.3.10.19
nvidia-cusolver-cu12     11.7.5.82
nvidia-cusparse-cu12     12.5.10.65
nvidia-nccl-cu12         2.29.7
nvidia-nvjitlink-cu12    12.9.86
openai                   0.28.0
opt_einsum               3.4.0
optree                   0.19.0
packaging                26.0
pip                      26.0.1
propcache                0.4.1
protobuf                 7.34.0
pydantic                 2.12.5
pydantic_core            2.41.5
Pygments                 2.19.2
requests                 2.32.5
rich                     14.3.3
setuptools               82.0.1
six                      1.17.0
sniffio                  1.3.1
tensorflow               2.21.0
termcolor                3.3.0
tqdm                     4.67.3
typing_extensions        4.15.0
typing-inspection        0.4.2
urllib3                  2.6.3
wheel                    0.46.3
wrapt                    2.1.2
yarl                     1.23.0
Relevant log output
(tf_test) user@user-Standard-PC-Q35-ICH9-2009:/home/lhj$ python callChainBuild/src_tensorflow/output_tensorflow/targeted_mutation/validation_sharded_full_20260322/potential_bugs/tf_raw_ops_WriteScalarSummary/0.py
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1774265664.422049 3732488 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1774265664.492746 3732488 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1774265665.815049 3732488 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1774265667.333059 3732488 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9606 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
I0000 00:00:1774265667.335750 3732488 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 9606 MB memory:  -> device: 1, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:02:00.0, compute capability: 7.5
I0000 00:00:1774265667.338406 3732488 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 9606 MB memory:  -> device: 2, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:03:00.0, compute capability: 7.5
I0000 00:00:1774265667.341101 3732488 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 9606 MB memory:  -> device: 3, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:04:00.0, compute capability: 7.5
E0000 00:00:1774265667.602828 3732488 util.cc:131] oneDNN supports DT_HALF only on platforms with AVX-512. Falling back to the default Eigen-based implementation if present.
Successfully wrote summary for value: 0.0
Successfully wrote summary for value: 0.0
Successfully wrote summary for value: 0
Successfully wrote summary for value: 0
Successfully wrote summary for value: 0.0
Segmentation fault (core dumped)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.