The training did not converge in the custom dataset

I have a custom dataset of 4000 rows of data. BUT the training did NOT converge (epoch=100,lr=0.001)
Here is my config:
```yaml
TRAIN:
ENABLE: True
DATASET: ava
BATCH_SIZE: 8
EVAL_PERIOD: 1
CHECKPOINT_PERIOD: 2
AUTO_RESUME: True
CHECKPOINT_FILE_PATH: /data/zhangshuang/workspace/SlowFast/checkpoints/SLOWFAST_4x16_R50.pkl
CHECKPOINT_TYPE: caffe2
DATA:
NUM_FRAMES: 32 # α * T = 8 * 4
SAMPLING_RATE: 2 #τ
TRAIN_JITTER_SCALES: [256, 320]
TRAIN_CROP_SIZE: 224
TEST_CROP_SIZE: 224
# TEST_CROP_SIZE: 256
INPUT_CHANNEL_NUM: [3, 3]
PATH_TO_DATA_DIR: '/data/zhangshuang/workspace/myAVA/Dataset'
DETECTION:
ENABLE: True
ALIGNED: True
AVA:
BGR: False
DETECTION_SCORE_THRESH: 0.5
FRAME_DIR: /data/zhangshuang/workspace/myAVA/Dataset/rawframes
FRAME_LIST_DIR: /data/zhangshuang/workspace/myAVA/Dataset/frame_lists
ANNOTATION_DIR: /data/zhangshuang/workspace/myAVA/Dataset/annotations
TRAIN_GT_BOX_LISTS: ["train.csv"]
TRAIN_PREDICT_BOX_LISTS: []
TEST_PREDICT_BOX_LISTS: [val.csv]
EXCLUSION_FILE: train_excluded_timestamps.csv
LABEL_MAP_FILE: action_list.pbtxt
GROUNDTRUTH_FILE: val.csv
SLOWFAST:
ALPHA: 8
BETA_INV: 8
FUSION_CONV_CHANNEL_RATIO: 2
FUSION_KERNEL_SZ: 7
RESNET:
ZERO_INIT_FINAL_BN: True
WIDTH_PER_GROUP: 64
NUM_GROUPS: 1
DEPTH: 50
TRANS_FUNC: bottleneck_transform
STRIDE_1X1: False
NUM_BLOCK_TEMP_KERNEL: [[3, 3], [4, 4], [6, 6], [3, 3]]
SPATIAL_DILATIONS: [[1, 1], [1, 1], [1, 1], [2, 2]]
SPATIAL_STRIDES: [[1, 1], [2, 2], [2, 2], [1, 1]]
NONLOCAL:
LOCATION: [[[], []], [[], []], [[], []], [[], []]]
GROUP: [[1, 1], [1, 1], [1, 1], [1, 1]]
INSTANTIATION: dot_product
POOL: [[[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]]]
BN:
USE_PRECISE_STATS: False
NUM_BATCHES_PRECISE: 200
SOLVER:
BASE_LR: 0.001
MOMENTUM: 0.9
WEIGHT_DECAY: 1e-7
OPTIMIZING_METHOD: sgd
MAX_EPOCH: 100
MODEL:
NUM_CLASSES: 8
ARCH: slowfast
MODEL_NAME: SlowFast
LOSS_FUNC: bce
DROPOUT_RATE: 0.5
HEAD_ACT: sigmoid
TEST:
ENABLE: False
DATASET: ava
BATCH_SIZE: 8
DATA_LOADER:
NUM_WORKERS: 4
PIN_MEMORY: True
NUM_GPUS: 1
NUM_SHARDS: 1
RNG_SEED: 0
OUTPUT_DIR: /data/zhangshuang/workspace/SlowFast/output/SLOWFAST_4*16_R50_epoch100_lr0.001

GPU: 0
TENSORBOARD:
ENABLE: True
LOG_DIR: /data/zhangshuang/workspace/SlowFast/output/SLOWFAST_4*16_R50_epoch100_lr0.001/log
CLASS_NAMES_PATH: /data/zhangshuang/workspace/SlowFast/configs/AVA/action_list.json
CONFUSION_MATRIX:
ENABLE: True
SUBSET_PATH: /data/zhangshuang/workspace/SlowFast/configs/AVA/action_list.txt
```
and the loss curve is shown in the figure