Add num_readers flag to eval_image_classifier.py

Prerequisites

• I checked to make sure that this feature has not been requested already.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/slim/eval_image_classifier.py

2. Describe the feature you request

I'd like to add num_readers flag to the evaluation script eval_image_classifier.py so that we can evaluate models using parallel dataset readers.

3. Additional context

The training script train_image_classifier.py has num_readers flag while eval_image_classifier.py does not have any counterpart.

Adding num_readers will not only make the help messages for the train/eval scripts more symmetric, but also actually speed up evaluation.

To see this, I ran some experiments on ImageNet with MobileNet_v1_1.0_224 as follows.

1. Added num_readers flag to my own copy of eval_image_classifier.py:

   +tf.app.flags.DEFINE_integer(
   +    'num_readers', 4,
   +    'The number of parallel readers that read data from the dataset.')
   

        provider = slim.dataset_data_provider.DatasetDataProvider(
            dataset,
            shuffle=False,
   +        num_readers=FLAGS.num_readers,
            common_queue_capacity=2 * FLAGS.batch_size,
            common_queue_min=FLAGS.batch_size)
   

2. Ran the modified version of eval_image_classifier.py

   • with validation and train split
   • with num_readers=1 (the same setting as in the current master branch) and num_readers=4
   • with num_preprocessing_threads=4 (default)

   and measured their respective ellapsed time.

   dataset_split_name	num_readers	num_preprocessing_threads	ellapsed time (sec)	gain (%)
   'validation'	1	4	86	-
   'validation'	4	4	78	9.30%
   'train'	1	4	1,678	-
   'train'	4	4	1,815	7.55%

   The results above suggest that incrementing num_readers can make evaluation actually faster.

3. Ran the same modified version of eval_image_classifier.py

   • with the training dataset
   • with num_readers={1,4,8}
   • with num_preprocessing_threads={4,8}

   and measured their respective ellapsed time (not all possible cominations were tested).

   dataset_split_name	num_readers	num_preprocessing_threads	ellapsed time (sec)	gain (%)
   'train'	1	4	1,678	-
   'train'	1	8	1,425	21.5%
   'train'	4	4	1,815	7.55%
   'train'	8	8	1,389	23.5%

   These results suggest that, in combination with num_preprocessing_threads flag, the evaluation on the whole ImageNet training dataset can be made up to 23.5% faster. Also, this speedup is not achieved by incrementing num_preprocessing_threads alone.

System information

• OS Platform and Distribution: CentOS Linux 7
• Mobile device name if the issue happens on a mobile device: N/A
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.15.3
• Python version: 3.7.6
• Bazel version (if compiling from source): N/A
• GCC/Compiler version (if compiling from source): N/A
• CUDA/cuDNN version: 10.0.130.1/7.6.5
• GPU model and memory: Tesla V100 with 16160MiB memory

4. Are you willing to contribute it?

Yes!