google-research / google-research/connectomics

what's the meaning of the naming convention for TFRecord files

Open
#86 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
66
Forks
17
Avg merge
6h 58m
Merged PRs (30d)
4

Description

Hi, long time no see. I encountered some confusion while studying your cool work.

When I was running the program[Train a SegCLR embedding model](https://colab.research.google.com/gist/chinasaur/d7139adf9fbb8df3f55a00afde2c32fe/segclr-embedding-training.ipynb) in
[SegCLR wiki](https://github.com/google-research/connectomics/wiki/SegCLR),
I ran the following code:
```python
tfrecord_pathspec = 'gs://h01-release/data/20230118/training_data/c3_positive_pairs/goog14c3_max200000_skip50.tfrecord@1000'
tfrecord_files = data_input_util.expand_pathspec(tfrecord_pathspec)
```
The above statement indicates that the program will load the training samples from Google Cloud [h01-release](https://console.cloud.google.com/storage/browser/h01-release/data/20230118/training_data/c3_positive_pairs?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false).

Perhaps as a beginner in TensorFlow2, I don't quite understand the meaning of the sample file name `goog14c3_max200000_skip50.tfrecord-00000-of-01000` in Google Cloud Storage. For example, what does `max200000` mean and what does `skip50` signify? It seems that `0000-of-01000` indicates that this is the `0th` file out of `1000` samples, because there are a total of 1000 TFRecord files in that directory. I was surprised to find that they all seem to be around `1.7G` in size. Does that mean that each TFRecord file represents randomly sampled pair information from a segment? So in `SegCLR`, a total of 1000 segments were collected from H01, and the number of pairs sampled from each segment was the same, resulting in the basic consistency of each TFRecord's size.

There may be some misunderstandings in my understanding. Please help me identify them.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.