deepinsight / deepinsight/insightface

[BUG] Malformed COCO RLE counts can overflow the vendored pycocotools decode buffer

Open
#2,889 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29.7k
Forks
6.1k
PR merge metrics
No merged PRs in 30d

Description

## Describe the bug

The vendored COCO mask implementation under `detection/retinaface/rcnn/pycocotools` accepts compressed RLE objects whose decoded run lengths do not match the declared mask size.

`_mask.decode(...)` allocates the destination mask from the untrusted `size` field:

```python
h, w, n = Rs._R[0].h, Rs._R[0].w, Rs._n
masks = Masks(h, w, n)
rleDecode(Rs._R, masks._mask, n)
```

But `rleFrString(...)` decodes the untrusted `counts` string into `R->cnts` without checking that the run-length total equals `h * w`. `rleDecode(...)` then trusts `R->cnts[j]` and writes that many bytes to the destination pointer:

```c
for( k=0; k ann["segmentation"]["counts"] and ann["segmentation"]["size"]
-> COCO.annToMask(...)
-> maskUtils.decode(rle)
-> pycocotools.mask.decode(...)
-> Cython extension _mask.decode(...)
-> _frString(...) / rleFrString(...)
-> R->cnts
-> rleDecode(...)
-> heap out-of-bounds write
```

This is at least a reliable process crash/DoS for code that decodes untrusted COCO RLE masks, and it is native heap memory corruption rather than a normal Python-level parsing error.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.