tensorflow / tensorflow/tensorboard

Weird num_thresholds behaviour in pr_curve plugin.

Open
#1,536 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

core:summaries plugin:pr-curves
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

TensorBoard: v1.11.0 according to pip, also 1.11.0 when running.
TensorFlow: v1.11.0
Python: 3.7.0

The num_thresholds parameter of pr_curve_raw_data_pb from the pr_curve plugin has inconsistent and weird behaviour.

  1. If num_thresholds < len(precision), then instead of approximating the PR curve in num_thresholds points, like one might expect, only the first num_thresholds points are taken. This cuts off the PR curve. There seems to be no reason to ever supply num_thresholds < len(precision), because this results in broken PR curves.

  2. The solution to (1) would be passing num_thresholds=len(precision). However if pr_curve_raw_data_pb is called multiple times (i.e. to get the steps slider), it seems, that only the first num_thresholds is respected and all the consecutive PR curves are cut off.

Example:

from tensorboard.summary import pr_curve_raw_data_pb
import tensorflow as tf
import numpy as np

# [1] Curve 1 (3 data points)
pre1 = np.asarray([0, .75, 1])
rec1 = np.asarray([1, .75, 0])
tp1 = fp1 = tn1 = fn1 = np.asarray([1] * 3)

# [2] Curve 2 (5 data points)
pre2 = np.asarray([0, .7, .8, .9, 1])
rec2 = np.asarray([1, .75, .5, .25, 0])
tp2 = fp2 = tn2 = fn2 = np.asarray([1] * 5)

with tf.summary.FileWriter('./logs') as writer:
    # [3] Plotting curve 1 with proper num_thresholds = len(precision) = 3
    problem2_1 = pr_curve_raw_data_pb('problem2', tp1, fp1, tn1, fn1, pre1, rec1, num_thresholds=len(pre1))
    writer.add_summary(problem2_1, 1)

    # [4] Plotting curve 2 with the same name as [3] with proper num_thresholds = len(precision) = 5
    problem2_2 = pr_curve_raw_data_pb('problem2', tp2, fp2, tn2, fn2, pre2, rec2, num_thresholds=len(pre2))
    writer.add_summary(problem2_2, 2)

The above code results in this PR curve for step 2 (only the first 3 points are drawn).
image

The full curve should look like this:
image

A solution would be either to allow different steps of the same curve to have different num_thresholds or to pick num_thresholds evenly spread samples from precision and recall arrays.

P.S. Also it would be great, if the true_negative parameter was optional. The concept of a True Negative doesn't exist in some tasks, like detection.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pr_curve_raw_data_pb entry point and reproduce the two calls in the issue using the provided Python example. Trace how num_thresholds is handled across repeated steps and verify the behavior for arrays of different lengths. Done means consecutive curves are not unexpectedly truncated, with regression coverage for the reported cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, tensorflow
Domain
data-visualization
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.