Whats the way to set up TF- Semantic Segmantation Pipeline for custom dataset properly? Getting weird training results

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [x] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### My Question

Hi everyone,
i am running the tf- semantic segmentation pipeline with my own pointcloud dataset. It only consists of 2 classes foreground/background(first picture). I got 1000 sample point clouds (800 for training, 200 for validation). The pipeline is unchanged, just editet the configfile(see below).

For any reason the iou is alternating between 0.5 and 1 while accuracy = 1 and Loss = 0.

Can anyone detect a reason for the behaviour or explain what happens? The data is balanced
How should i set up the pipeline to run correctly?

The config file looks like following:
``` dataset:
name: SegmentCar
dataset_path: /home/datasets/
train_folder: dataset_00009
test_folder: Test
cache_dir: /home/logs/cache
ignored_label_inds: []
test_result_folder: ./test
use_cache: False
sampler:
name: 'SemSegRandomSampler'
model:
name: RandLANet
batcher: DefaultBatcher
ckpt_path: # path/to/your/checkpoint
num_neighbors: 16
num_layers: 5
num_points: 65536
num_classes: 2
ignored_label_inds: [0]
sub_sampling_ratio: [4, 4, 4, 4, 2]
in_channels: 5
dim_features: 8
dim_output: [16, 64, 128, 256, 512]
grid_size: 0.06
weight_decay: 0.001
augment:
recenter:
dim: [0, 1, 2]
pipeline:
name: SemanticSegmentation
num_workers: 0
optimizer:
lr: 0.001
batch_size: 2
learning_rate: 0.01
main_log_dir: /home/logs/checkpoints
max_epoch: 100
save_ckpt_freq: 20
scheduler_gamma: 0.9886
test_batch_size: 1
train_sum_dir: /home/logs/tensorboard
val_batch_size: 2
summary:
record_for: []
max_pts:
use_reference: false
max_outputs: 1
```