As a developer, I want a resource to understand the work to transition the pipeline from direct LLM calls to DSPy.

# Context

As we finish laying down the fondation for the transition of our pipeline from direct LLM calls to DSPy, we now look into documenting the work done so far to help facilitate the transfer of knowledge. This documentation will hopefully provide a structured overview of DSPy, the work we did, the challenges we faced, our results and future improvements.

# Suggested Documentation Outline

## Contextualizing DSPy
(Minimal contextualization of DSPy with links to relevant official documentation)

### The Main Moving Parts of a DSPy Program
- **Signature**: Defines input -> output behavior
- **Predictor/Built-in Modules**: The strategy used to solve the task
- **Metrics**: A numerical representation of the quality of the output
- **Optimizers**: Built-in tools to optimize the model toward a specific behavior

### The Main Iterative Loop of Creating a DSPy Module
1. Defining the task and the overall module input and output
2. Defining the initial pipeline and the data flow
3. Finding a bunch of examples to test the program against
4. Defining an initial metric to assess what "good" looks like
5. Analyzing the results of the evaluation and identifying areas where the module struggles
6. Attempting to solve pain points using various vectors of improvement
7. Rinse and repeat

## The Work Done So Far
(Explanation of the various challenges that have been addressed or partially addressed)

### Setting Up a Project Structure
- Our goal was to create a structure that ensures modularity and reproducibility
- Overview of the project structure:
- Data
- Notebooks
- Scripts
- Pipeline
- Test data, etc.

### Defining the Task, the Main Inputs, Outputs, and the Pipeline
- Documenting the choices made for the signature and use of a monolithic module
- Documenting how the various components of the pipeline interact

### Curating a Dataset of Examples
- The core dataset existed, and we built on top of it
- To enable reproducibility, we created a script that generates a CSV of examples from raw data

### Defining the Metric Function
- Explanation of how the metric function drives iterative improvement and optimization
- Outlining different moving parts of our evaluation methodology (e.g., techniques used for different inspection fields)

### Optimization
- Rudimentary work done
- Explanation of the bottleneck due to limited data
- The few-shot optimizer was the only viable choice since we only had 35 examples
- With a 20/80 split, that left barely 7 training examples

### Data Flywheel:
- Automating data supplementation
- The idea is to scale our dataset by integrating user-generated entries from the deployed application
- This may or may not be completed by the end of the month if not we can move this to Next Steps

## The Next Steps
(Discussion on performance and potential improvements)

### System Performance
- Listing observed areas of failure

### Various Vectors of Improvement Moving Forward

#### Improving the Architecture
- Breaking down the monolithic module into sub-modules targeting different aspects of inspection creation
- Addressing identified failure points through specialized modules
- Example: If label information is lost during OCR, an OCR enhancement module could be created to preprocess results before classification

#### Optimizing Modules
- Three main types of optimization
1. Demonstration tuning
- More viable after breaking the monolithic module into sub-modules
- Monolithic structure leads to extremely long prompts
2. Holistic tuning (prompt + demonstration)
- MIPROv2 is a flagship optimizer in DSPy
- Likely a significant avenue for improvement once we have at least 200 training examples (or 1000 total with the suggested 20/80 split)
3. Weight tuning
- If we adopt open-weight models and collect a large dataset, weight optimization could be explored

#### Improving the Metric
- The metric function dictates the improvement direction
- As our understanding of the problem space grows, refining the evaluation metric should be considered