Pretrained models for Keypoint + Segmentation tasks

[saidaltindis]

Pretrained models for Keypoint + Segmentation tasks

Although there are multiple pretrained models provided for segmentation and keypoint tasks separately, there is no pretrained model that trained for multi-task.

Providing such models would be an important baseline for the people that re-trained their model since it is possible to train such models using detectron2 (#17).

[VGrondin]

The keypoints model are multi-task, you just need to turn it on e.g.:

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Keypoints/keypoint_rcnn_X_101_32x8d_FPN_3x.yaml"))
cfg.MODEL.MASK_ON = True
cfg.MODEL.KEYPOINT_ON = True 

[saidaltindis]

Are you sure? They did not report the mask score for keypoint models and in metrics, mask loss is not present.

https://dl.fbaipublicfiles.com/detectron2/COCO-Keypoints/keypoint_rcnn_R_50_FPN_1x/137261548/metrics.json

[chenzhutian]

After setting cfg.MODEL.MASK_ON = True, the output of the mask is just noise. I guess the pre-trained models dont support keypoint detection and segmentation in the same time.

[soulslicer]

Did you every figure this out?

[DIANSLEE]

I tried
cfg.MODEL.MASK_ON = True
cfg.MODEL.KEYPOINT_ON = True
and I combine the .pkl weight file both mask and keypoints to get pretrained weights.
however the segment AP is down 20 points....