tensorflow / tensorflow/tflite-support

Semantic Segementation Example "Label number 21 mismatch" exception

Open
#770 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
441
Forks
146
PR merge metrics
No merged PRs in 30d

Description

When Following the example form this page :
www.tensorflow.org/lite/examples/segmentation/overview
I downloaded the linked .tflite model LiteModelDeeplabv31Metadata2 and imported it into my Android Studio project
using the right-click -> new -> other -> Tensor Flow Lite Model option.
The IDE Recognized the model and gave me the following code example to use it :

LiteModelDeeplabv31Metadata2 model = LiteModelDeeplabv31Metadata2.newInstance(context);

// Creates inputs for reference.
TensorImage image = TensorImage.fromBitmap(bitmap);

// Runs model inference and gets result.
LiteModelDeeplabv31Metadata2.Outputs outputs = model.process(image);
List<Category> segmentationMasks = outputs.getSegmentationMasksAsCategoryList(); 

When trying to execute this i get the following exception:

    java.lang.IllegalArgumentException: Label number 21 mismatch the shape on axis 1
        at org.tensorflow.lite.support.common.internal.SupportPreconditions.checkArgument(SupportPreconditions.java:104)
        at org.tensorflow.lite.support.label.TensorLabel.<init>(TensorLabel.java:87)
        at org.tensorflow.lite.support.label.TensorLabel.<init>(TensorLabel.java:105)
        at de.tudortmund.mb.ips.ml.LiteModelDeeplabv31Metadata2$Outputs.getSegmentationMasksAsCategoryList(LiteModelDeeplabv31Metadata2.java:108)

Looking through the stacktrace it becomes obvious what is going wrong :
getSegmentationMasksAsCategoryList(LiteModelDeeplabv31Metadata2.java:108)
is from the automatically generated code and
it creates a new new TensorLabel and calls .getCategoryList() on it.
However the constructor already fails:

This is the called Constructor :

public TensorLabel(@NonNull List<String> axisLabels, @NonNull TensorBuffer tensorBuffer) {
    this(makeMap(getFirstAxisWithSizeGreaterThanOne(tensorBuffer), axisLabels), tensorBuffer);
}

It receives a List of Labels as strings form the included meta data (21 strings),
creates a hashmap from it using the makeMap method with getFirstAxisWithSizeGreaterThanOne(tensorBuffer)
as its only key and the label list as the it's value.
This is strange however as the label dimension is not the FirstAxisWithSizeGreaterThanOne
but the last axis as per documentation.
The shape of the tensorBuffer is (1,257,257,21) where 257*257 is the resolution of the 21 semantic masks
which is the correct format according to this site:
www.tensorflow.org/lite/inference_with_metadata/task_library/image_segmenter

This wrongly created hashmap causes the actual constructor to fail in the last checkArgument call :

public TensorLabel(
      @NonNull Map<Integer, List<String>> axisLabels, @NonNull TensorBuffer tensorBuffer) {
    SupportPreconditions.checkNotNull(axisLabels, "Axis labels cannot be null.");
    SupportPreconditions.checkNotNull(tensorBuffer, "Tensor Buffer cannot be null.");
    this.axisLabels = axisLabels;
    this.tensorBuffer = tensorBuffer;
    this.shape = tensorBuffer.getShape();
    for (Map.Entry<Integer, List<String>> entry : axisLabels.entrySet()) {
      int axis = entry.getKey();
      SupportPreconditions.checkArgument(
          axis >= 0 && axis < shape.length, "Invalid axis id: " + axis);
      SupportPreconditions.checkNotNull(entry.getValue(), "Label list is null on axis " + axis);
      SupportPreconditions.checkArgument(
          shape[axis] == entry.getValue().size(),
          "Label number " + entry.getValue().size() + " mismatch the shape on axis " + axis);
    }
}

Either the called TensorLabel constructor has to be changed or the hashmap has to be created in advance and then the right
constructor can be called immediately (the one receiving a Map not a List)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading TensorLabel.java, including the List and Map constructors, then inspect the generated LiteModelDeeplabv31Metadata2.java entry point named in the stack trace. Reproduce the semantic-segmentation example with the documented (1,257,257,21) tensor and confirm the label axis is handled without the mismatch exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, java, tensorflow
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.