DDMAL / DDMAL/mothra-annotator
[Discussion - problem?] YOLO detection is not splitting neumes properly, we may need to retrain
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 0
- Avg merge
- 45m
- Merged PRs (30d)
- 2
Description
When identifying neumes in the IC, we mostly work with the smallest neume/neume components possible, so that the neumes that the IC is trained to identify are as concise and generalizable as possible. This is particularly relevant in square notation, where compound neumes can get pretty big. For example, a neume like this:
We would want to split like this:
And a neume like this:
We would want to split like this:
This allows us to classify the first neume as `neume.podatus2a`, `neume.oblique2`, and `neume.torculus33`, and the second as two `neume.podatus2b` and two `neume.inclinatum`. These are all elements that are found in most--if not all--square notation manuscripts.
As per @Yueqiao12Zhang's [comment](https://github.com/DDMAL/Standalone-Interactive-Classifier/issues/93#issuecomment-5168340557), this splitting action has to happen in the YOLO layer detection stage. However, when the YOLO detection model was first trained, large compound neumes like these were put in one, big bounding box. This means that YOLO almost never splits neumes into small enough components, and compound neumes end up in the IC as is and aren't classified properly. This is particularly problematic for obliques, which are almost always attached to at least one punctum.
To be very clear, YOLO is not over-splitting neumes, it's massively under-splitting them. If we want to retrain YOLO to split neumes properly (or even just a little), I think we'll need to modify the ground truth for neumes pretty significantly, because I never split neumes when creating the original ground truth.
This issue has already been discussed [here](https://github.com/DDMAL/Standalone-Interactive-Classifier/issues/93) and [here](https://github.com/DDMAL/Standalone-Interactive-Classifier/issues/78#top); future discussions can be continued in this current issue, to keep everything in the same place.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue points to the YOLO layer detection stage and the existing neume ground truth; start by reviewing the linked discussions in issues 93 and 78 and how their proposed splitting behavior is represented. Determine the revised annotation scope and retraining process, with done meaning compound neumes are split into the stated components and classified correctly.
Written by the indexing model from the issue text.
Assessment
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100