tensorflow / tensorflow/tensorboard
Weird num_thresholds behaviour in pr_curve plugin.
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
TensorBoard: v1.11.0 according to pip, also 1.11.0 when running.
TensorFlow: v1.11.0
Python: 3.7.0
The num_thresholds parameter of pr_curve_raw_data_pb from the pr_curve plugin has inconsistent and weird behaviour.
-
If
num_thresholds < len(precision), then instead of approximating the PR curve innum_thresholdspoints, like one might expect, only the firstnum_thresholdspoints are taken. This cuts off the PR curve. There seems to be no reason to ever supplynum_thresholds < len(precision), because this results in broken PR curves. -
The solution to (1) would be passing
num_thresholds=len(precision). However ifpr_curve_raw_data_pbis called multiple times (i.e. to get the steps slider), it seems, that only the firstnum_thresholdsis respected and all the consecutive PR curves are cut off.
Example:
from tensorboard.summary import pr_curve_raw_data_pb
import tensorflow as tf
import numpy as np
# [1] Curve 1 (3 data points)
pre1 = np.asarray([0, .75, 1])
rec1 = np.asarray([1, .75, 0])
tp1 = fp1 = tn1 = fn1 = np.asarray([1] * 3)
# [2] Curve 2 (5 data points)
pre2 = np.asarray([0, .7, .8, .9, 1])
rec2 = np.asarray([1, .75, .5, .25, 0])
tp2 = fp2 = tn2 = fn2 = np.asarray([1] * 5)
with tf.summary.FileWriter('./logs') as writer:
# [3] Plotting curve 1 with proper num_thresholds = len(precision) = 3
problem2_1 = pr_curve_raw_data_pb('problem2', tp1, fp1, tn1, fn1, pre1, rec1, num_thresholds=len(pre1))
writer.add_summary(problem2_1, 1)
# [4] Plotting curve 2 with the same name as [3] with proper num_thresholds = len(precision) = 5
problem2_2 = pr_curve_raw_data_pb('problem2', tp2, fp2, tn2, fn2, pre2, rec2, num_thresholds=len(pre2))
writer.add_summary(problem2_2, 2)
The above code results in this PR curve for step 2 (only the first 3 points are drawn).

The full curve should look like this:

A solution would be either to allow different steps of the same curve to have different num_thresholds or to pick num_thresholds evenly spread samples from precision and recall arrays.
P.S. Also it would be great, if the true_negative parameter was optional. The concept of a True Negative doesn't exist in some tasks, like detection.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the pr_curve_raw_data_pb entry point and reproduce the two calls in the issue using the provided Python example. Trace how num_thresholds is handled across repeated steps and verify the behavior for arrays of different lengths. Done means consecutive curves are not unexpectedly truncated, with regression coverage for the reported cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, tensorflow
- Domain
- data-visualization
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100