tensorflow / tensorflow/tensorboard
PR-curves plugin (pr_curve_raw_data_op) not plotting completely when using Estimator API
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
TensorBoard version (from pip package, also printed out when running tensorboard): 1.12.0
TensorFlow version if different from TensorBoard: 1.12.0
OS Platform and version (e.g., Linux Ubuntu 16.04): Arch Linux
Python version (e.g. 2.7, 3.5): 3.6.5
Hi there,
I am trying to plot PR curves using pr_curve_raw_data_op function with precisions, recalls and other metrics which I have calculated before hand, but I'm getting wrong results in chart. I am using Estimator API, meaning I create my own dictionary which contains tuple of tf.Tensor and tf.Operation instances which are then forwarded to estimator for dumping summaries.
I evaluate multiple steps during each checkpoint save, which means that I accumulate precision and recall through multiple steps (but I don't use pr_curves plugin to do so, I use custom written metrics), and all I want from the plugin is to dump the PR curve into the tensorboard. Here is the part of the code which serves for storing the pr-curves summary tensor into the metrics dictionary:
metrics_dict['pr_curve/CLASS_{}'.format(cls_index)] = (
summary.raw_data_op(
'pr_curve/CLASS_{}'.format(cls_index),
true_positive_counts=tps,
false_positive_counts=fps,
true_negative_counts=tns,
false_negative_counts=fns,
precision=precisions,
recall=recalls,
num_thresholds=11
),
tf.no_op()
)
I also tried to dump precisions as text summary just to be sure that output printed in terminal is the same that is being stored in summaries, and it was I dumped those summaries using this code:
metrics_dict['precisions_text/CLASS_{}'.format(cls_index)] = (
tf.summary.text(name="aa",
tensor=tf.as_string(precs)),
tf.no_op()
)
In the end I will attach couple of screenshots. First one shows precisions dumped to text summary and their values ( I only showed first 6 values, as for the remaining 4 values are 0.0):

Second screenshot contains those same precisions for same class and step plotted on PR curve:

You can see that there is only first point on the plot, with precision being 1.0 for 0.0 recall. That occurs in several more cases, along with chart cutting off at last point (for example, I have precision values from recalls 0.0 up to 0.6, but chart plots only up to 0.5).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue through the pr_curve_raw_data_op and summary.raw_data_op entry points using the Estimator API and the supplied precision, recall, and count tensors. Compare the values in the text summary with the PR-curves plugin output; done means every supplied threshold point is plotted with the expected precision and recall, including the final points.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-visualization
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100