Adding multiple classification heads to train

I am trying to build a model with multiple classification heads. MaskRCNN already has 3 heads - mask, box and class
How can we add another custom head such as another classification head. Is there a possibility to extend the current implementation of detectron to cater to this problem statement?

Say for example I have a problem statement in which I am supposed to infer the location, bbox, class, color of fruit from an image.
For inferring the class and the color one way can be to club the two into one class making a combination of each class with each color but that creates somewhat imbalanced datasets. Can I rather do multiple classification heads where one predicts the class and the other predicts the color of the crop?

Start by reading the existing MaskRCNN implementation and tracing how its mask, box, and class heads are connected during training. Then identify how a second classification target for attributes such as fruit color would be represented; the issue names no files or tests, so the required design and validation approach would need to be established first.

Written by the indexing model from the issue text.