How does IR face recognition work with a model trained on RGB?
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 412
- PR merge metrics
- No merged PRs in 30d
Description
Sorry, this might not strictly qualify as an issue but it feels like the documentation is a bit lacking in this context and I don't see any other place to ask...
I've been trying to make sense out of how Howdy works on the inside but I am struggling a bit.
As far as I can tell the DLIB model used by `howdy` has been trained on RGB images of people (https://github.com/davisking/dlib-models#dlib_face_recognition_resnet_model_v1datbz2) but howdy is supposed to use images taken with an IR webcam. The `VideoCapture` class seems to return both an RGB frame as well as a grayscale frame (https://github.com/boltgolt/howdy/blob/beta/howdy/src/recorders/video_capture.py#L97) which is confusing since OBS is giving me what I would call grayscale when using my IR camera (where is the RGB coming from?). Howdy then proceeds to use the grayscale frame for face detection (https://github.com/boltgolt/howdy/blob/beta/howdy/src/cli/add.py#L171) but the RGB frame for getting the actual encoding of the face (https://github.com/boltgolt/howdy/blob/beta/howdy/src/cli/add.py#L201) which feels quite sketchy since it would defeat the whole thing with the IR camera...
Could someone please explain this to me? What am I missing?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with howdy/src/recorders/video_capture.py around line 97, then trace the grayscale and RGB frames through howdy/src/cli/add.py around lines 171 and 201. Compare those paths with the linked dlib RGB model documentation and document how IR input is handled and whether the encoding path is valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- authentication, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100