Training object detection models cause CPU soft lockup

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/object_detection

2. Describe the bug

Whenever starting training, and whether it's successful or not, I get the following kernel issue:

Message from syslogd@cgpu1 at Jan 20 12:01:56 ...
kernel:[4247215.993987] watchdog: BUG: soft lockup - CPU#43 stuck for 23s! [model_main_tf2.:944304]

3. Steps to reproduce

Initiate training using the following command:
python3 model_main_tf2.py --model_dir= --pipeline_config_path=

4. Expected behavior

Training starts and finishes without issues

5. Additional context

Message from syslogd@cgpu1 at Jan 20 12:02:24 ...
kernel:[4247243.993987] watchdog: BUG: soft lockup - CPU#43 stuck for 23s! [model_main_tf2.:944304]

Message from syslogd@cgpu1 at Jan 20 12:03:00 ...
kernel:[4247279.993987] watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [model_main_tf2.:944304]

Message from syslogd@cgpu1 at Jan 20 12:03:32 ...
kernel:[4247311.989985] watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [model_main_tf2.:944304]

Message from syslogd@cgpu1 at Jan 20 12:04:00 ...
kernel:[4247339.989985] watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [model_main_tf2.:944304]

Message from syslogd@cgpu1 at Jan 20 12:04:32 ...
kernel:[4247371.989986] watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [model_main_tf2.:944304]

Message from syslogd@cgpu1 at Jan 20 12:05:16 ...
kernel:[4247415.993988] watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [model_main_tf2.:944304]

Message from syslogd@cgpu1 at Jan 20 12:05:44 ...
kernel:[4247443.993988] watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [model_main_tf2.:944304]

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04.3 LTS
• Mobile device name if the issue happens on a mobile device:
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): v2.7.0-rc1-69-gc256c071bb2
• Python version: Python 3.8.10
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: 10.1, V10.1.243
• GPU model and memory: Quadro RTX 8000 48 GB