It takes time to load cached images

I have a StatefulWidget that represents a page in my app where we show a few images from network using CachedNetworkImage.
I've found out it takes time to load some images while it's instant for some others ([video recording](https://photos.google.com/share/AF1QipP_Hb8dAT2b9qlSDsHDiE1Q4g3k8XzC9KtkI_8lHHOY1Rbx5xYEHbp-EPG9FuxssA/photo/AF1QipOJuUYwx8FsWYxsVLZ9Fd15asRla8fUgFdVAY70?key=cGNSeDdJM19rQ3pxTVRTNlhoeDh4RVdTTTJOVWNB)) even though they have been all cached with this CacheManager (a singleton one)

```
class CustomImageCacheManager {
static const key = 'imageCacheManager';
static CacheManager instance = CacheManager(
Config(
key,
stalePeriod: const Duration(days: 7),
maxNrOfCacheObjects: 20,
),
);
}
```

When I looked at the cache files for the app, I saw the cached images and their cache info. Example:
```
{
"url": "https://images.unsplash.com/photo-1663640171543-9eef40a4b320?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=687&q=80",
"key": "https://images.unsplash.com/photo-1663640171543-9eef40a4b320?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=687&q=80",
"relativePath": "08663510-57b0-11ed-8310-89bb5cdfb17a.jpg",
"eTag": null,
"validTill": 1982424908512,
"touched": 1667064908546,
"length": 58360,
"_id": 11
},
```

Is this an expected behavior? Is there any way to make the rendering instant for all cached images?