ProjectSidewalk / ProjectSidewalk/RampNet
Add YOLOE (real-time open-vocab) to the zero-shot detector bucket
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7
- Forks
- 1
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 7
Description
Motivation
The open-vocab bucket (#39 / PR #40) currently holds OWLv2 and Grounding DINO, both 2024-era. YOLOE ("Real-Time Seeing Anything," Tsinghua, Mar 2025) is the current SOTA real-time open-vocab detector — +3.5 AP over YOLO-Worldv2 on LVIS at ~1.4× speed, with text / visual / prompt-free modes. Adding it (a) modernizes the bucket, (b) adds a real-time speed axis against the slow VLMs, (c) tests whether a newer open-vocab model is a more efficient recall complement than OWLv2.
Expectation-setting (from PR #40)
The "purpose-built open-vocab beats chat VLMs" hypothesis was refuted — OWLv2 / gdino did worse (best-sweep F1 owlv2 0.184 vs gemini-3.6 0.634 vs rampnet 0.855); text-prompted open-vocab isn't selective enough for curb ramps. So YOLOE will most likely lose on F1 too. Its real value is currency + the recall-oracle angle: OWLv2 recovered 69/72 of RampNet's misses (union recall 0.990) but at 36–128 FP per recovered ramp — a 6–20× less efficient complement than Gemini (#35). Open question: does YOLOE's visual-prompt / prompt-free mode complement RampNet more efficiently?
What to run
Add a yoloe provider to compare.py, text prompt "curb ramp" / "wheelchair ramp" (plus try visual-prompt mode with a few ramp exemplars). Feed via the existing reprojection rig; boxes → center points → matcher at 0.022. Calibrated scores → AP / PR / --sweep like the other open detectors. Low effort — Ultralytics-packaged, no new eval path.
Caveats
Honor the nadir/hood masking follow-up (#47 — a third of each view is vehicle hood), and the parser/rig cache-key gap (bust .model_cache on any parser change).
Relates to #39 (open detectors) and #20 (harness). Companion supervised baseline: #51.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in compare.py and trace the existing open-vocabulary providers through the reprojection rig, box-to-center conversion, matcher threshold 0.022, and calibrated AP/PR and --sweep evaluation. Add the YOLOE provider with the specified text prompts and investigate visual-prompt mode using a few ramp exemplars. Done means YOLOE runs through the existing path, produces comparable metrics, and respects nadir/hood masking and cache invalidation after parser changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100