dandi / dandi/s3invsync

Include retry logic and logging for scenario: "0 B/s was observed" error

Open
#176 6 comments 0 reactions 0 assignees View on GitHub
error handling
Dominant language
Rust
Stars
3
Forks
1
PR merge metrics
No merged PRs in 30d

Description

While running an `s3invsync` backup, the follow error was observed:

```
Error: failed downloading contents for s3://linc-brain-mit-prod-us-east-2/blobs/290/e07/290e070d-9681-41d4-a0f0-966df911e943

Caused by:
0: streaming error
1: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed
```

While it is hard to trace whether underlying infrastructure may have caused zero bytes to be transferred at that time, `s3invsync` may be a much more robust tool if retry logic was in place for this error state

The logs show this error occurring 3 times, prior to the script shutting down.

For confirmation, the given asset does exist in the S3 bucket, and is of size > 0B:

```
aaronkanzer@Aarons-MacBook-Pro ~ % aws s3 ls s3://linc-brain-mit-prod-us-east-2/blobs/290/e07/290e070d-9681-41d4-a0f0-966df911e943 --human-readable --summarize
2024-03-13 23:46:46 7.5 MiB 290e070d-9681-41d4-a0f0-966df911e943

Total Objects: 1
Total Size: 7.5 MiB
```

**Note**: Upon re-running the script, this asset was properly backed up.

Cc @kabilar @yarikoptic @jwodder

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.