Batch Image Cropping

Nobody has claimed this yet.

We need an efficient, parallel pipeline to process a large dataset of images stored in Parquet files. The pipeline will run on a SLURM cluster.

**Workflow:**
* **Input:** A large number of Parquet files or image folder containing coded images
* **Process:** Read, decode, and crop images using detection results.
* **Output:** Store the resulting cropped images in HDF5 files (to avoid filesystem I/O bottlenecks from millions of PNGs).

---

### **Architecture Decision & Thought Process**

**Chosen Approach:** SLURM Job Array + Internal Python Scheduler

Initially considered two options:
1. A SLURM Job Array with simple, single-core tasks.
2. A hybrid model using fewer, "big" SLURM tasks with more cores and memory, each managing its own `multiprocessing` pool. Kinda like Spark...

We have decided to proceed with **Option 1**.

* **Rationale:** We are prioritizing simplicity and robustness. We will keep the core Python processing script **single-threaded**! This avoids the complexities of managing a `multiprocessing.Pool` (like worker management, data serialization, and error handling).
* **Scaling Strategy:** We will use SLURM's job array feature (`sbatch --array...`) as the primary mechanism for scaling. This is the most "SLURM-native" way to handle an "embarrassingly parallel" problem and allows the scheduler to efficiently pack our jobs across the cluster.

---

### **Implementation Tasks**

1. **Simple Python Processor:**
* Create the core Python script that performs the read, decode, crop, and HDF5-write logic for a *batch* of images. This script should remain single-threaded.
* Enable Streaming feature from Arrow to avoid OOM from reading everything all at once

2. **Internal "Scheduler Class":**
* Develop a class *within* the Python script that is responsible for work assignment.
* This class will read the `$SLURM_ARRAY_TASK_ID` environment variable.
* Based on this ID and the total number of array tasks (`$SLURM_ARRAY_TASK_COUNT`), it will determine *which* specific Parquet files (or chunks of data) this individual task is responsible for processing.

3. **HDF5 Output:**
* Define the output schema for the HDF5 files.
* Each SLURM task should probably write to its own unique HDF5 file (e.g., `output_part_${SLURM_ARRAY_TASK_ID}.h5`) to prevent file-locking conflicts.

4. **Master `sbatch` Script:**
* Create the final `sbatch` script that launches the job array.
* It will specify the resource allocations (`--ntasks=1`, `--cpus-per-task=1`, `--mem-per-cpu`, etc.) and call the main Python script.

Start by locating the repository's existing Python inference or data-processing entry points and any SLURM submission conventions. Then define the single-threaded processor, the scheduler class using SLURM_ARRAY_TASK_ID and SLURM_ARRAY_TASK_COUNT, per-task HDF5 output, and the master sbatch script; done means the full batch workflow is specified and runnable on the cluster.

Written by the indexing model from the issue text.