Performance on minority classes

I ran the inference on a video where there was shooting but it seems that the trained models don't really have good performance to predict shoot which is one the classes and instead picks "carry/hold (an object)". Not sure whether I am doing something wrong or that is expected due to data imbalance etc. (also tried on train data but it gets classified as "carry/hold (an object)"). Would appreciate any comments/help. The models I tried were pre-trained model for AVA dataset and for the config I have:

DEMO:
ENABLE: True
FPS: 30
OUTPUT_FILE: ./demo_ms_32.mp4
LABEL_FILE_PATH: ./ava_classids.json # Add local label file path here.
INPUT_VIDEO: demo/movie_shoot.mkv
DETECTRON2_CFG: "COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"
DETECTRON2_WEIGHTS: detectron2://COCO-Detection/faster_rcnn_R_50_FPN_3x/137849458/model_final_280758.pkl
COMMON_CLASS_NAMES: []