The test performance of Toronto3D with TF version is strange, how to solve?

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### My Question

I have run semantic segmentation on Toronto3D dataset with TF version, but the test result is strange. It's too low. Have you ever met this problem? Thanks in advance.

Started testing
running inference
Accuracy : [0.988935467546158, 0.00642939150401837, 0.5693296344345207, 0.7759625851802733, 0.9279876765324961, 0.5254333985970463, 0.5190694823267895, 0.003669624274290722, 0.5396021575494492]
IoU : [0.9416579999000307, 0.006429000736093571, 0.5622588740268236, 0.37500095701315644, 0.24888122187286138, 0.4320609873120802, 0.40914981001954714, 0.002926530968812789, 0.3722956727311757]
Per class Accuracy : [0.988935467546158, 0.00642939150401837, 0.5693296344345207, 0.7759625851802733, 0.9279876765324961, 0.5254333985970463, 0.5190694823267895, 0.003669624274290722]
Per class IOUs : [0.9416579999000307, 0.006429000736093571, 0.5622588740268236, 0.37500095701315644, 0.24888122187286138, 0.4320609873120802, 0.40914981001954714, 0.002926530968812789]
Overall Accuracy : 0.540
Overall IOU : 0.372