Profile the predict_tile method.

The predict_tile function is the workhorse of the entire package. Most people will arrive with large geospatial tiles that cannot fit into memory. The predict_tile function needs to be more thoroughly profiled and understood.

General instructions and psuedo-code

1. Take a large tile from https://zenodo.org/record/5912107#.ZBiLR-zMKDU

wget https://zenodo.org/record/5912107/files/2018_TEAK_3_315000_4094000_image_crop.tif?download=1 .

2. Use the predict tile function to generate predictions

3. Post the cProfile results here and identify which facets are slow under the following four conditions, cpu/gpu with config["workers"] > 0, 2) and cpu/gpu with config["workers"] == 0. What number of workers is fastest? Please post full details of operating system and GPU.

The motivation was I was running predict_tile on a large number of RGB tiles on SLURM linux setup with GPU with 10 workers. I'm seeing alot of variation in the run time among tiles. This makes sense when there are no trees predicted, since there will be no non-max suppression to run (which suggests NMS is costly), but beyond that the iterations/per second seem to vary wildly.

One tile 389 crops took 30 min

Predicting DataLoader 0: 100%|██████████| 389/389 [30:40<00:00,  4.73s/it]

Another tile 729 crops took 23 seconds just a minute later. Something is going on here with the workers locking?

100%|██████████| 729/729 [00:23<00:00, 30.65it/s]

Start by locating the predict_tile entry point and download the large RGB tile from the linked Zenodo record. Run it under CPU and GPU with config["workers"] > 0 and == 0, collecting cProfile results for each condition. Done means reporting the slow facets, fastest worker count, operating system, and GPU details.