mlcommons / mlcommons/tiny

Loss is negative and accuracy=0.006 when tried to prune anamoly detection

Open
#118 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
479
Forks
116
PR merge metrics
No merged PRs in 30d

Description

I have used Tensorflow optimization toolkit to prune the benchmark anamoly_detection.

The procedure I have followed is same as shown in the below link.
https://www.tensorflow.org/model_optimization/guide/combine/pqat_example

The output during training is like this:
Epoch 2/100
2412/2412 [==============================] - 20s 8ms/step - loss: 11.1539 - val_loss: 11.1307
Epoch 3/100
2412/2412 [==============================] - 20s 8ms/step - loss: 10.6982 - val_loss: 10.6691
Epoch 4/100
2412/2412 [==============================] - 20s 8ms/step - loss: 10.4117 - val_loss: 10.5804
Epoch 5/100
2412/2412 [==============================] - 20s 8ms/step - loss: 10.2858 - val_loss: 10.2876
Epoch 6/100
2412/2412 [==============================] - 20s 8ms/step - loss: 10.1822 - val_loss: 10.2884
Epoch 7/100
2412/2412 [==============================] - 20s 8ms/step - loss: 10.1250 - val_loss: 10.2690
Epoch 8/100
2412/2412 [==============================] - 20s 8ms/step - loss: 10.0805 - val_loss: 10.3325

Output during pruning is like this:
Epoch 1/100
2680/2680 [==============================] - 47s 16ms/step - loss: -291053.6562 - accuracy: 0.0359
Epoch 2/100
2680/2680 [==============================] - 42s 16ms/step - loss: -287242.3438 - accuracy: 0.0335
Epoch 3/100
2680/2680 [==============================] - 42s 16ms/step - loss: -294022.2500 - accuracy: 0.0341
Epoch 4/100
2680/2680 [==============================] - 41s 15ms/step - loss: -301931.7188 - accuracy: 0.0336
Epoch 5/100
2680/2680 [==============================] - 41s 15ms/step - loss: -311050.6875 - accuracy: 0.0294
Epoch 6/100
2680/2680 [==============================] - 42s 15ms/step - loss: -321004.3750 - accuracy: 0.0241
Epoch 7/100
2680/2680 [==============================] - 41s 15ms/step - loss: -331941.9375 - accuracy: 0.0177
Epoch 8/100
2680/2680 [==============================] - 42s 16ms/step - loss: -343348.4688 - accuracy: 0.0109
Epoch 9/100
2680/2680 [==============================] - 42s 16ms/step - loss: -355611.3438 - accuracy: 0.0080
Epoch 10/100
2680/2680 [==============================] - 42s 16ms/step - loss: -368586.7812 - accuracy: 0.0073
Epoch 11/100
2680/2680 [==============================] - 42s 16ms/step - loss: -382228.8125 - accuracy: 0.0068
Epoch 12/100
2680/2680 [==============================] - 42s 16ms/step - loss: -396485.6875 - accuracy: 0.0066
Epoch 13/100
2680/2680 [==============================] - 42s 16ms/step - loss: -411286.4062 - accuracy: 0.0065
Epoch 14/100
2680/2680 [==============================] - 41s 15ms/step - loss: -426653.8750 - accuracy: 0.0061
Epoch 15/100
2680/2680 [==============================] - 41s 15ms/step - loss: -442495.9062 - accuracy: 0.0056
Epoch 16/100
2680/2680 [==============================] - 41s 15ms/step - loss: -458785.0625 - accuracy: 0.0049

Can you help me with this issue?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the anomaly_detection pruning procedure described in the issue and compare it with the linked TensorFlow Model Optimization Toolkit PQAT example. Trace the training and evaluation setup to identify why pruning produces negative loss and very low accuracy; done means a confirmed cause and a documented fix or reproducible diagnosis.

Written by the indexing model from the issue text.

Assessment

Tech stack
tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.