Scalability Considerations + Plan [WIP]

Goal - generate embeddings over Kenya for 8 years with the S2 encoder.

Some rough notes about a low engineering effort for scaling out to Kenya size. Suggestions + insights and additional considerations are very welcome!

Tiling:

- Kenya is covered by ~91 MGRS tiles (note there are some overlaps, and the perimeter tiles cover majority ocean and should be addressed)

**Estimated Input Data Size for Kenya**:

- 1 MRGS tile with the 10 Sentinel 2 bands for 1 year is ~29Gi * 8 (years) * 91 (total MGRS tiles that cover Kenya) ~= 2-3TB

**Estimated Embedding Data Output Size**

~20 TB 8 years all of Kenya
~185 GB for per MGRS tile for 8 years
~ 37 GB for the priority areas within Kenya from the labeled dataset shared by the academic team (since the output sizes are so large let's first prioritize this to share with the academic team then think about generating more of Kenya)

**Potential Compute Approach**

* Scale with AWS batch (using spot instances), we could have a batch job per each year and tile and spin up a dask cluster per batch job to further parallelize the processing within each MGRS tile.

* It is important to note that the MGRS tiles do overlap, so this will have implications for our plan of writing the embeddings out, the overlap will most likely cause issues if we're trying to write chunk-wise into a single zarr file that covers the extent of Kenya.

- [ ] Next steps to confirm if this is reasonable - try to generate embeddings for a few MGRS tiles using the EC2 instance we have access to via Fields of the World compute. This instance is a a T3 with ~8cpu + ~32gi

**Open Questions**

* Do we need GPU or can inference be run on a CPU?

testing locally on a laptop indicates that generating the embeddings is memory limited, not I/O limited, so we can do some additional tests and get better time estimates to see how long CPU only process would take

* Do we need to persist the input S2 data?
Probably depends if we're going to use GPU for inference or not, but if we're going to run with CPU, it would be ideal to not write the data out

cc @yellowcap for a heads up about size estimates, I'll update with time estimates after testing on the EC2.