Sync performance improvements

Hi!

I am considering using turso in an app. Right now it uses remote-only which is fine but I'd like the DB to be used offline so naturally reached for offline reads with read-your-writes. The problem is, since each frame is very small, every time you open the app you probably need to sync:

• 0 or 1 generations
• 0 - 250 (or so) frames

The generation happens in a single request. The frames however take significantly longer. The issue is twofold:

• there is no way to find out what the greatest frame number is for a given generation
• due to this, the client fetches them one-at-a-time
• that also includes 125 average-case round-trip requests which, depending on where you are (say, SF <-> EU) means 13-14s just waiting for light to travel

For my use-case (app loaded once a week that requires a reasonable loading time) waiting over a minute is not a reasonable tradeoff for offline access.

Suggestion:

• the ideal solution would be a single API call that can get you up-to-date from any generation
  • /sync/from/{local_generation}/{frame}
  • if /info shows that current_generation - local_generation > 1 we prefix the response with an entire sqlite db
  • if /info shows that current_generation - local_generation == 1 we yield all the remaining frames in local_generation joined with all the frames in current_generation
    • this assumes that sending at most 500 frames will be cheaper than 1 whole db and 250 frames, however depending on the size of your db (and once you add in the 4kb minimum frame size) this assumption may not hold
  • otherwise, if they are equal, we just include all the remaining frames in current_generation
• if a one-size-fits-all endpoint doesn't cut it, then we can do 3 requests
  • add max_frame_no parameter to /info
  • client decides whether to pull the latest generation
  • optimise /sync/{current_generation}/{frame}/{max_frame_no} to fetch more efficiently (reuse connections, parallelize fetch)
    • S3 can handle 5500 concurrent GET requests so (assuming the backend is in rust) joining all the futures is perfectly valid
• another option is to have /sync/1/1/200 try to fetch every single frame requested (1-200) in parallel but, if a frame doesn't exist (say, max_frame_no = 30) just return the successful data.
  • Presumably there is some crate that will take a stream of ordered futures that resolve to Option<Result<Frame, Err>> and:
    • when encountering an option, cancel all the 'later' futures since they will also be 404
    • when encountering an actual error, return early and yield the error
  • S3 will still bill for 404 errors

Relevant thread https://discord.com/channels/933071162680958986/1360940737977516154/1361025506451001364

Start by reading the existing /info and /sync endpoints and the linked Discord discussion. Compare the proposed single-call and three-request approaches against current frame and generation behavior, including offline reads and-read-your-writes. Done means an agreed sync design with clear API semantics and an implementation plan.