tursodatabase / tursodatabase/libsql

If bottomless misses uploading a single frame, then restoration will fail

Open
#686 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bottomless nemesis
Dominant language
C
Stars
17.2k
Forks
531
Avg merge
1h 12m
Merged PRs (30d)
1

Description

Lets say there is an network issue and bottomless failed to upload a single frame, then sent rest of the frames, then restoration can fail. Snapshot at shutdown fixes this issue, but if the DB did not have a clean shutdown then it won't work.

To reproduce this issue (you can use Nemesis):

  1. Run sqld and insert a few rows in a db
  2. Stop local minio server
  3. Insert a few more rows
  4. Resume minio and insert few more rows
  5. Kill the sqld (i.e. unclean shutdown)
  6. On restart, the restoration would fail

Possible solution:

We write all the frames to disk before uploading to S3. We could probably build a local queue and when an upload is successful, it should check if there are any previously unsent frames present on disk and upload them.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the bottomless frame upload, local-disk queue, and restoration paths; the issue does not name specific files or tests. Reproduce the failure with the listed MinIO outage and unclean shutdown steps, then verify that a missed frame is uploaded later and restoration succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.