test_schema_meta_validation[maskrcnn_resnet50_fpn_v2] needs to be fixed
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 7.3k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 13
Description
This test is randomly failing:
Traceback (most recent call last):
File "/home/circleci/project/test/test_extended_models.py", line 276, in test_schema_meta_validation
assert not incorrect_meta
AssertionError: assert not [(MaskRCNN_ResNet50_FPN_V2_Weights.COCO_V1, '_ops')]
Due to this check:
Turns out the get_ops() function is sensitive to the random seed which leads to different computed OPS and makes the test fail. I don't know if it's get_ops() who's wrong, or if there is something inherently buggy in MaskRCNN. Or if this is expected and we just need to set the random seed. IDK.
We have to skip this test for now to unblock the release, but we should go back to it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in test/test_extended_models.py, especially test_schema_meta_validation around lines 265-276, and reproduce the failure for MaskRCNN_ResNet50_FPN_V2. Trace how get_ops() produces the _ops metadata and compare runs with different random seeds. Done means the test reliably validates the model metadata without requiring the temporary skip.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100