google-deepmind / google-deepmind/dmvr

Inconsistent label mapping among train, test, and validation sets

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
68
Forks
14
PR merge metrics
No merged PRs in 30d

Description

Hi,

First of all, thanks for sharing this code.

I am using it for implementing some Video Vision Transformers with a custom dataset. I wrote my own code for generating the CSV files, following the example you provided for the HMDB dataset. However, I have noticed an inconsistent behavior in [generate_from_file.py](https://github.com/deepmind/dmvr/blob/master/examples/generate_from_file.py) when generating the TFRecord shards:

https://github.com/deepmind/dmvr/blob/77ccedaa084d29239eaeafddb0b2e83843b613a1/examples/generate_from_file.py#L179-L183

If I understand well, [set](https://www.w3schools.com/python/python_sets.asp#:~:text=Set%20is%20one%20of%204,items%20and%20add%20new%20items.) removes duplicate values but, at the same, it is unordered, so you cannot be sure in which order the items will appear. This means that, even having the same labels for training, validating, and testing in my CSV files, there is a chance that the label mapping is different when generating the TFRecord shards of each subset. I have done a small check of this issue by printing `l_map`, and these are the mappings I got for each subset (using the same 4 labels in all the subsets and with no modifications in `generate_from_file.py`):

```
.../dataset/DMVR_train.csv
{'Label_C': 0, 'Label_A': 1, 'Label_D': 2, 'Label_B': 3}
.../dataset/DMVR_test.csv
{'Label_A': 0, 'Label_C': 1, 'Label_D': 2, 'Label_B': 3}
.../dataset/DMVR_validation.csv
{'Label_B': 0, 'Label_D': 1, 'Label_C': 2, 'Label_A': 3}
```

In fact, the mapping is different for each subset whenever I rerun this small test. I am not sure if there might be an error in my CSV files (I do not really think so, but maybe) or just a particular issue with my custom dataset.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.