DDMAL / DDMAL/mothra-annotator

[Discussion - problem?] YOLO detection is not splitting neumes properly, we may need to retrain

Open
#23 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
0
Forks
0
Avg merge
45m
Merged PRs (30d)
2

Description

When identifying neumes in the IC, we mostly work with the smallest neume/neume components possible, so that the neumes that the IC is trained to identify are as concise and generalizable as possible. This is particularly relevant in square notation, where compound neumes can get pretty big. For example, a neume like this:

Image

We would want to split like this:

Image

And a neume like this:

Image

We would want to split like this:

Image

This allows us to classify the first neume as `neume.podatus2a`, `neume.oblique2`, and `neume.torculus33`, and the second as two `neume.podatus2b` and two `neume.inclinatum`. These are all elements that are found in most--if not all--square notation manuscripts.

As per @Yueqiao12Zhang's [comment](https://github.com/DDMAL/Standalone-Interactive-Classifier/issues/93#issuecomment-5168340557), this splitting action has to happen in the YOLO layer detection stage. However, when the YOLO detection model was first trained, large compound neumes like these were put in one, big bounding box. This means that YOLO almost never splits neumes into small enough components, and compound neumes end up in the IC as is and aren't classified properly. This is particularly problematic for obliques, which are almost always attached to at least one punctum.

To be very clear, YOLO is not over-splitting neumes, it's massively under-splitting them. If we want to retrain YOLO to split neumes properly (or even just a little), I think we'll need to modify the ground truth for neumes pretty significantly, because I never split neumes when creating the original ground truth.

This issue has already been discussed [here](https://github.com/DDMAL/Standalone-Interactive-Classifier/issues/93) and [here](https://github.com/DDMAL/Standalone-Interactive-Classifier/issues/78#top); future discussions can be continued in this current issue, to keep everything in the same place.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue points to the YOLO layer detection stage and the existing neume ground truth; start by reviewing the linked discussions in issues 93 and 78 and how their proposed splitting behavior is represented. Determine the revised annotation scope and retraining process, with done meaning compound neumes are split into the stated components and classified correctly.

Written by the indexing model from the issue text.

Assessment

Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.