Network IO bottleneck on cache server

One thing we encountered implementing a similar system inspired by your work @XiangpengHao is IO bottlenecking: since many compute nodes are hitting the same cache node(s) and those cache nodes then have to hit object storage all of the IO has to go through the cache nodes. This is fine as long as you don't max out IO on the cache nodes but in our production workloads we hit this pretty quickly.

┌───────────┐                                                   
│           │                                                   
│ Compute 1 ├──────┐                                            
│           │      │                                            
└───────────┘      │                                            
                   │                                            
┌───────────┐      │       ┌───────────┐           ┌───────────┐
│           │      └───────▶           ├───────────▶  Object   │
│Compute ...├──────────────▶   Cache   ├───────────▶  Storage  │
│           │      ┌───────▶           ├───────────▶           │
└───────────┘      │       └───────────┘           └───────────┘
                   │                                            
┌───────────┐      │                                            
│           │      │                                            
│ Compute N ├──────┘                                            
│           │                                                   
└───────────┘                                                   

Is there something that LiquidCache does differently to avoid this?

What I was thinking for our system is that the cache needs to only process cached data. Basically it should return an enum of Hit(Data) or Miss(GoDoTheWorkYourself) and record cache misses so it can populate itself async according to some cache policy.

No files, tests, or entry points are named. Start by tracing LiquidCache's cache-node and object-storage path, then determine the intended cache-miss behavior and measurable completion criteria; the issue currently does not define what a finished change should demonstrate.