🐛 Bug: IndexError in Heuristic Pitch Finding Due to Inconsistent Staff Line Data

_(I tagged some folks in the assignees field more or less just to get eyes here who might/will be interested, no obligation of course to do anything with/about this)_ 😁

## Issue Description

When running the workflow to produce MEI files with stave and glyph information, the `heuristic pitch finding` job fails with an `IndexError: list index out of range` error. This occurs specifically when working with historical manuscripts that have imperfections such as curved lines, faded ink, spotty quill marks, and background noise.

### Error Message

```
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "/code/Rodan/rodan/jobs/base.py", line 791, in run
retval = self.run_my_task(inputs, settings, arg_outputs)
File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/base.py", line 183, in run_my_task
pitches = pf.get_pitches(glyphs, staves)
File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/PitchFinding.py", line 45, in get_pitches
self._find_pitches(self.glyphs)
File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/PitchFinding.py", line 104, in _find_pitches
line_or_space, line_num = self._return_line_or_space_no(g, center_of_mass, staff_locations)
File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/PitchFinding.py", line 405, in _return_line_or_space_no
pb_left = line[line_pos[0]]
IndexError: list index out of range
```

## Root Cause Analysis

After investigation, the root cause appears to be inconsistencies in the staff line data structure produced by the staff finding job. Specifically:

1. The staff finding algorithm is too rigid in handling real-world manuscript characteristics
2. It produces JSOMR files with inconsistent staff line data structures
3. The pitch finding algorithm expects consistent line data structures but fails when encountering inconsistencies

### Example of Problematic Staff Line Data

In the JSOMR file, some staff lines have fewer points than others. For example, in staff_no 4:

```json
"line_positions": [
[[1204, 1402], [1435, 1412], [1579, 1415], [1722, 1417], [1723, 1417]],
[[1204, 1434], [1435, 1444], [1579, 1447], [1722, 1449], [1723, 1449]],
[[1204, 1466], [1435, 1476], [1579, 1479], [1722, 1481], [1723, 1481]],
[[1203, 1499], [1579, 1507], [1722, 1511]], // Fewer points here!
// ... more lines with inconsistent point counts
]
```

When the pitch finding algorithm iterates through these lines to calculate glyph positions, it tries to access `line[line_pos[0]]` where `line_pos[0]` is an index available in previous lines but not in the current line, resulting in the `IndexError`.

## Contributing Factors

The staff finding algorithm struggles with manuscripts that have:

1. **Background page noise** - Creates false positives or breaks in legitimate line detection
2. **Curved staff lines** - Due to either the original manuscript or page bending during photography
3. **Faded ink** - Creates artificial breaks in what should be continuous lines
4. **Spotty quill/ink marks** - Causes inconsistent line thickness and density

These factors, especially when occurring simultaneously, make it difficult for the current rigid algorithm to maintain consistent data structures.

## Reproduction Steps

1. Use an input manuscript with the characteristics described above (faded lines, curves, noise)
2. Run the `miyao staff finding` job to produce a JSOMR file
3. Use this JSOMR file along with an XML file containing classified glyphs as input to the `heuristic pitch finding` job
4. Observe the `IndexError` when processing glyphs that fall between lines with inconsistent point counts

## Verification & Current Workaround

When using a JSOMR file with consistent staff line data (produced using polygon data converted to JSOMR), the workflow succeeds, confirming that the issue lies in the staff line data structure.

### Working Workaround

A functional workaround has been identified:
1. Instead of using `miyao staff finding`, use `miyao staff finder` which produces polygon files
2. Convert the polygon file to JSOMR format
3. Use this converted JSOMR with the `heuristic pitch finding` job
4. This passes without errors and correctly proceeds to the MEI generation step

This workaround confirms that the issue is with the JSOMR structure produced by `miyao staff finding`, not with the `heuristic pitch finding` job itself (when given properly structured input).

## Proposed Solutions

### Current Workaround

Use the following alternative workflow that bypasses the issue:
1. Use `miyao staff finder` instead of `miyao staff finding` (this produces polygon data)
2. Convert the polygon file to JSOMR format
3. Use this converted JSOMR with the `heuristic pitch finding` job
4. Continue to MEI generation

This workaround has been tested and successfully produces the desired results without errors.

### Short-term Fixes

1. **Add error handling in the pitch finding algorithm**: Modify `_return_line_or_space_no` to handle cases where line data is incomplete:

```python
# Before trying to access line[line_pos[0]]
if line_pos[0] >= len(line):
# Option 1: Skip this glyph or provide a default position
return 0, len(staff) # Default value

# Option 2: Interpolate the missing point based on available data
# interpolated_point = interpolate_point(line, line_pos[0])
# pb_left = interpolated_point
else:
pb_left = line[line_pos[0]]
```

2. **Create a pre-processing step** between staff finding and pitch finding that normalizes the JSOMR data to ensure consistent line structures.

3. **Document the proven workaround** in the official documentation to help other users encountering this issue.

### Long-term Improvements

1. **Improve the staff finding algorithm** to be more robust with historical manuscripts:
- Increase tolerance for variance in staff line detection
- Implement adaptive thresholding based on local image conditions
- Use curve-fitting models rather than straight-line heuristics
- Ensure consistent sampling across all lines

2. **Refactor the pitch finding algorithm** to be more resilient:
- Implement proper error handling for incomplete data
- Support interpolation for missing points
- Make fewer assumptions about the consistency of input data

## the Polygon Processing Script: Tolerance and Grouping
**Polygon Processing Script Summary**
- The script implements a system for processing polygons representing musical notation elements with these key components:
- Vertical Tolerance and Grouping
- The script uses a vertical proximity-based clustering approach with a configurable vertical_tolerance parameter (default set to 100 pixels). - Polygons are first sorted by their top-most y-coordinate, then grouped together if their vertical centers are within this tolerance of the group's average center. This creates initial staff groups based on vertical positioning.

#### Boundary Procedure
For each grouped staff, the script:

1. Calculates a bounding box by finding the minimum/maximum x and y coordinates across all points in the group
2. Validates the bounding box dimensions (skipping groups with invalid dimensions)
3. Visualizes each staff group with color-coded rectangles showing the bounding box
4. Uses the bounding box coordinates for the JSOMR output format

#### Line Following/Interpolation
The script implements a sophisticated line following procedure:

1. Each staff is divided into 8 horizontal lines based on vertical anchors
2. For each line, it:
- Identifies candidate points within a vertical band around the estimated line position
- Filters points to ensure unique x-coordinates for interpolation
- Performs linear interpolation to generate evenly spaced points along the line
- Falls back to equally-spaced points if insufficient candidates are found
3. Generates 100 points per line (configurable) across the staff width

This approach handles non-straight staff lines by following the actual polygon contours rather than using simple straight lines, making it robust for warped or curved notation staves in manuscript images.

This procedure effectively transforms polygon data into properly structured staff notation with appropriate vertical spacing and horizontal continuity.

@DeannaLC I saw that you used a similar approach for #1254 addressing the spottiness and bending of staves on a page; just wanted to tag you additionally given these modified approaches work, though I started from a different path than you did. `Miyao Staff Finder` also has more settings which can be altered than `Finding`, if you get curious with tinkering.

## Files
Polygons
`Miyao Staff Finding` produced JSOMR
polygons2jsomr script
grouping visualization from script
fixed JSOMR which works in `Heuristic Pitch Finding`
Original PNG (Chorale107)
Staff Layer sample (Staff layer from JSB_Chorale_0107).
This link ([here](https://mcgill-my.sharepoint.com/:f:/g/personal/kyrie_bouressa_mail_mcgill_ca/EmCTEZlrvjZAp01TAQDGuQ8BCW68tgqzsXnOJVK4z_hB6A?e=tAHYWW)) will have everything. If you cannot access them, just holler.

These items can be run through a stripped down e2e workflow, wherein only the second branch is present. You can position `Finder` above `Finding` (make sure you add png w/ miyao results as a mask as an export so you can see what `Finder` is seeing; you can then tinker extensively. I'm working with five line staves, but have found setting lines/staves to 4.