[Object Detection] autoaugment causes training to crash

System information

• What is the top-level directory of the model you are using: object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): v1.12.0-0-ga6d8ffae09 1.12.0
• Bazel version (if compiling from source):
• CUDA/cuDNN version: 9.0, 7.3.1
• GPU model and memory: Nvidia Tesla K80, 12GB memory
• Exact command to reproduce:

Describe the problem

I'm training an object detection model using SSD mobilenet v2 (COCO pretrained) using the model_main.py training script and wanted to use autoaugment for data preprocessing. Training runs fine with other data augmentation strategies, but autoaugment (with the "v0" policy) produces a couple different errors:

Traceback (most recent call last):
  File "/home/david_staub/.conda/envs/user/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in_do_call
    return fn(*args)
  File "/home/david_staub/.conda/envs/user/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in_run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/david_staub/.conda/envs/user/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in_call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Paddings must be non-negative: -1 0
         [[{{node cond_22/cond/cond/while/cond/Pad}} = Pad[T=DT_UINT8, Tpaddings=DT_INT32, _device="/device:CPU:0"](cond_22/cond/cond/while/cond/Slice, cond_22/cond/cond/while/cond/PadV2/paddings)]]
         [[{{node IteratorGetNext}} = IteratorGetNext[output_shapes=[[24], [24,300,300,3], [24,2], [24,3], [24,100], [24,100,4], [24,100,36], [24,100,36], [24,100], [24,100], [24,100], [24]], output_types=[DT_INT32, DT_FLOAT, DT_INT32, DT_INT32,DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]
         [[{{node GroupCrossDeviceControlEdges_0/ConstantFolding/Loss/Compare_7/IOU/Intersection/split_1-folded-3/_10252}}= _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_3293_...1-folded-3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

or

  File "/home/david_staub/.conda/envs/user/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in_do_call
    return fn(*args)
  File "/home/david_staub/.conda/envs/user/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in_run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/david_staub/.conda/envs/user/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in_call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: output dimensions must be positive
         [[{{node cond_10/cond/cond/while/cond/translate/transform/ImageProjectiveTransformV2}} = ImageProjectiveTransformV2[dtype=DT_UINT8, interpolation="NEAREST", _device="/device:CPU:0"](cond_10/cond/cond/while/cond/translate/transform/strided_slice, cond_10/cond/cond/while/cond/translate/translations_to_projective_transforms/concat, cond_10/cond/cond/while/cond/translate/transform/strided_slice_2)]]
         [[{{node IteratorGetNext}} = IteratorGetNext[output_shapes=[[24], [24,300,300,3], [24,2], [24,3], [24,100], [24,100,4], [24,100,36], [24,100,36], [24,100], [24,100], [24,100], [24]], output_types=[DT_INT32, DT_FLOAT, DT_INT32, DT_INT32,DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_BOOL, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]
         [[{{node GroupCrossDeviceControlEdges_0/ConstantFolding/Loss/Compare_7/IOU/Intersection/split_1-folded-3/_10252}}= _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_3293_...1-folded-3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

The errors seems to be non-deterministic: when I restart training it runs fine for different numbers of steps before crashing again.