ProjectSidewalk / ProjectSidewalk/RampNet

Add YOLOE (real-time open-vocab) to the zero-shot detector bucket

Open
#52 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
7
Forks
1
Avg merge
4d 11h
Merged PRs (30d)
7

Description

Motivation

The open-vocab bucket (#39 / PR #40) currently holds OWLv2 and Grounding DINO, both 2024-era. YOLOE ("Real-Time Seeing Anything," Tsinghua, Mar 2025) is the current SOTA real-time open-vocab detector — +3.5 AP over YOLO-Worldv2 on LVIS at ~1.4× speed, with text / visual / prompt-free modes. Adding it (a) modernizes the bucket, (b) adds a real-time speed axis against the slow VLMs, (c) tests whether a newer open-vocab model is a more efficient recall complement than OWLv2.

Expectation-setting (from PR #40)

The "purpose-built open-vocab beats chat VLMs" hypothesis was refuted — OWLv2 / gdino did worse (best-sweep F1 owlv2 0.184 vs gemini-3.6 0.634 vs rampnet 0.855); text-prompted open-vocab isn't selective enough for curb ramps. So YOLOE will most likely lose on F1 too. Its real value is currency + the recall-oracle angle: OWLv2 recovered 69/72 of RampNet's misses (union recall 0.990) but at 36–128 FP per recovered ramp — a 6–20× less efficient complement than Gemini (#35). Open question: does YOLOE's visual-prompt / prompt-free mode complement RampNet more efficiently?

What to run

Add a yoloe provider to compare.py, text prompt "curb ramp" / "wheelchair ramp" (plus try visual-prompt mode with a few ramp exemplars). Feed via the existing reprojection rig; boxes → center points → matcher at 0.022. Calibrated scores → AP / PR / --sweep like the other open detectors. Low effort — Ultralytics-packaged, no new eval path.

Caveats

Honor the nadir/hood masking follow-up (#47 — a third of each view is vehicle hood), and the parser/rig cache-key gap (bust .model_cache on any parser change).


Relates to #39 (open detectors) and #20 (harness). Companion supervised baseline: #51.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in compare.py and trace the existing open-vocabulary providers through the reprojection rig, box-to-center conversion, matcher threshold 0.022, and calibrated AP/PR and --sweep evaluation. Add the YOLOE provider with the specified text prompts and investigate visual-prompt mode using a few ramp exemplars. Done means YOLOE runs through the existing path, produces comparable metrics, and respects nadir/hood masking and cache invalidation after parser changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.