MLOps Model Training Pipeline

# Background

Current model training involves a set of steps requiring manual intervention. We need an observable semi automated process to track progress from raw image to final model and benchmarks. The process for Nachet mini will be prioritized and the process for the full platform will be handled separately at a later date.

***The process doesn't necessarily always start with the upload of new images. Models are based on datasets and datasets can be created from a combination of pre existing images***

## Process
1) Image Processing
- Raw training images uploaded to shared folder
- Import into cluster storage
- Image normalization
- sha 256 naming
- png conversion
- Images need to be organized depending on if the physical seeds were pre split into training and validation
- Validation images are separated out early
- The train test set are together and the split is handled by the trainer script
2) Image Labeling
- Create a COCO dataset file for the images
- Pre annotate bounding boxes with SAM3 or our own pre trained detector
- Convert data format to something compatible with FiftyOne
- Validate and fix annotations with Fiftyone
- Convert data format to something compatible with the trainer script (COCO)
3) Define a dataset
- This would either be just the set of new images or a combination of the new images with old images
4) Model Training
-
5) Model Benchmarking
- Run validation set
- Run benchmarks collected during end user system evaluation
-
6) Model Deployment
- Deploy to HuggingFace
- Update the model list
7) Observable Model Export
8) Observable Model Benchmarking
9) Observable Model Deployment
10) E2E tests
- Run playwright test against the deployed nachet-mini

## Action Items

### Infrastructure
We will use Argo Workflows for it's similarity to GitHub actions and Azure DevOps actions.
- [x] Deploy Argo
- [x] create an example workflow
- [x] create an example with a step requiring human intervention and approval

### Image Processing

- [ ] #895

### Model Training

- [ ]