facebookresearch / facebookresearch/detectron2
Low-memory object to anchor assignment method
- Dominant language
- Python
- Stars
- 34.7k
- Forks
- 7.9k
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Reduce memory consumption of anchor to object assignment
The current method for assigning objects to anchors might easily run out of GPU memory since it computes a _number of objects_ by _number of anchors_ score matrix. A reduction implemented within a cuda kernel could significantly reduce memory consumption allowing to use larger inputs with more objects.
## Motivation & Examples
I've used detectron2 in a task where I had to use a large number of anchors (over a million) and there were many small objects (could be over 2k in a sample). The default matching method which generates a matrix with pairwise iou values did not fit in GPU memory and the fallback method (do assignment on CPU tensors) was terribly slow.
To overcome this problem I've implemented a cython function to do the assignment in a memory-efficient way, without the need to sore the full iou matrix in memory.
However this method slowed down the process a bit (~10% in my setting), so I implemented a cuda kernel using numba to do the assignment. This showed no slowdown but significantly reduced memory usage (from 11.3GB to 3.9GB in my setup) thereby allowing to increase number of anchors and objects.
If this is something you would like to add to detectron2 I can create a pull request for it.
## Additional considerations
I have two concerns about the current implementation - adding anchors with maximal iou for each object:
1) I've found that there are quite often anchors with so similar iou (difference typically below 1e-6) that this must be due to rounding errors (anchor and object sizes were typically between 4..40px). I think filtering for maximal iou introduces noise in the assignment and therefore it seems beneficial to allow slightly lower ious as well (e.g. max-1e-5).
2) The anchor where an unassigned object is present with maximal iou is added as foreground while it is not sure that this particular object was assigned to that anchor in the first step.
3) Sometimes many anchors might match an object with the same iou, adding an inproportionally large number of low quality correspondences might set the training back.
Collected some statistics during training for my use case with many small objects:
number of assigned objects 61484
number of assigned anchors 75644
number of force-assigned objects (allow_low_quality_matches) 195455
number of force-assigned anchors (iou>=max-1e-5) 589836
number of indices changed during force-assignment 25936
number of force-assigned anchors (iou==max) 281767
In my case, allowing a small noise in iou roughly doubled the number of anchors which means that half of the corresponding anchors are discarded due to rounding errors and not geometrical considerations.
The number of anchors where a different object was assigned in the second step is rather small (4-5%) but if these are rather invalid assignments, I believe this could hold back the training.
Here the average number of assigned anchors per object is higher for low quality matches, filtering only the best (e.g. anchor whose center is closest to object's center) would make sense to me.
## Implementation
The way I implemented the new assignment is a new LowMemMatcher class derived from detectron2.modeling.matcher.Matcher. Also added a new field to the configuration MODEL.RPN.LOW_MEM_MATCHING, if it is true, this new class is instantiated in detectron2.modeling.proposal_generator.rpn.RPN and in this case the pairwise ious are not computed in RPN.label_and_sample_anchors but the gt_boxes and anchors are passed to the (LowMemMatcher) matcher object.
Contributor guide
Research direction
Start by reading detectron2.modeling.matcher.Matcher and the RPN paths in detectron2.modeling.proposal_generator.rpn.RPN, especially label_and_sample_anchors, along with the proposed MODEL.RPN.LOW_MEM_MATCHING configuration. Review the LowMemMatcher design described in the issue; done means supporting memory-efficient anchor assignment without materializing the full pairwise IoU matrix while addressing the listed low-quality matching concerns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100