deepinsight / deepinsight/insightface
[BUG] Malformed COCO RLE counts can overflow the vendored pycocotools decode buffer
- Dominant language
- Python
- Stars
- 29.7k
- Forks
- 6.1k
- PR merge metrics
- No merged PRs in 30d
Description
## Describe the bug
The vendored COCO mask implementation under `detection/retinaface/rcnn/pycocotools` accepts compressed RLE objects whose decoded run lengths do not match the declared mask size.
`_mask.decode(...)` allocates the destination mask from the untrusted `size` field:
```python
h, w, n = Rs._R[0].h, Rs._R[0].w, Rs._n
masks = Masks(h, w, n)
rleDecode(Rs._R, masks._mask, n)
```
But `rleFrString(...)` decodes the untrusted `counts` string into `R->cnts` without checking that the run-length total equals `h * w`. `rleDecode(...)` then trusts `R->cnts[j]` and writes that many bytes to the destination pointer:
```c
for( k=0; k ann["segmentation"]["counts"] and ann["segmentation"]["size"]
-> COCO.annToMask(...)
-> maskUtils.decode(rle)
-> pycocotools.mask.decode(...)
-> Cython extension _mask.decode(...)
-> _frString(...) / rleFrString(...)
-> R->cnts
-> rleDecode(...)
-> heap out-of-bounds write
```
This is at least a reliable process crash/DoS for code that decodes untrusted COCO RLE masks, and it is native heap memory corruption rather than a normal Python-level parsing error.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.