Backblaze Multi threaded upload will have a recoverable error - fails anyway
Open
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 59.9k
- Forks
- 5.4k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 48
Description
Continuation of:
https://github.com/rclone/rclone/issues/8315#issuecomment-2655173223
PR for fix: https://github.com/rclone/rclone/pull/8384
Almost like clockwork, rclone will fail at 3% for this file below. Even when the b2 concurrency value is set to just one.
@ncw mentioned that it looks like a proxy error with the HTML being sent back as an error code, or vice versa.
Looking at this logs, retry=False was being triggered for this error, borking the entire multi-part download consisting of 4k upload parts. Yes, if any one of those upload parts fails then the entire thing fails.
The reason is this: low-level retries are ONLY enabled if and only if, the error type and message can evaluate for true in list of predicate that will be run on the error. If no rule returns true, then the retries=False is returned.
This means that any unexpected error not covered by the white list will bypass the low level retries. That's what happened in the B2 case.
My fix does not address the bigger picture - it only adds this specific error type to the white list.
It may be useful to consider that retry=True be returned for any 500 "internal error" status code returned for an upload part.
650 failed: multi-thread copy: failed to write chunk: Post "https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0054": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 8.663 MiB/s, ETA 1w4d1h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m48.3s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.561Mi/s, 11h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 8.663 MiB/s, ETA 1w4d1h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m48.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.561Mi/s, 11h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 8.121 MiB/s, ETA 1w4d19h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m49.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.026Mi/s, 12h
2|run2 | 2025/02/12 20:47:59 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 25 (retry=false): Post "https://pod-020-3011-15.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0045": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-15.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0045", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 8.121 MiB/s, ETA 1w4d19h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m49.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.026Mi/s, 12h
2|run2 | 2025/02/12 20:47:59 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 25: Post "https://pod-020-3011-15.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0045": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 8.121 MiB/s, ETA 1w4d19h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m49.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.026Mi/s, 12h
2|run2 | 2025/02/12 20:47:59 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 19 (retry=false): Post "https://pod-020-3011-05.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0033": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-05.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0033", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 8.121 MiB/s, ETA 1w4d19h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m49.2s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.026Mi/s, 12h
2|run2 | 2025/02/12 20:47:59 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 19: Post "https://pod-020-3011-05.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0033": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 8.121 MiB/s, ETA 1w4d19h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m49.2s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.026Mi/s, 12h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 8.121 MiB/s, ETA 1w4d19h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m49.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.026Mi/s, 12h
2|run2 | 2025/02/12 20:48:00 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 109 (retry=false): Post "https://pod-020-3011-14.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0041": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-14.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0041", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.614 MiB/s, ETA 1w5d14h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m49.8s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.026Mi/s, 12h
2|run2 | 2025/02/12 20:48:00 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 109: Post "https://pod-020-3011-14.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0041": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.614 MiB/s, ETA 1w5d14h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m49.8s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 8.026Mi/s, 12h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.614 MiB/s, ETA 1w5d14h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m50.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.524Mi/s, 13h
2|run2 | 2025/02/12 20:48:00 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: multi-thread copy: chunk 52/11650 failed: multi-thread copy: failed to write chunk: Post "https://pod-020-3011-13.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0034": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.614 MiB/s, ETA 1w5d14h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m50.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.524Mi/s, 13h
2|run2 | 2025/02/12 20:48:00 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: multi-thread copy: chunk 185/11650 failed: multi-thread copy: failed to write chunk: Post "https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0044": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.614 MiB/s, ETA 1w5d14h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m50.3s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.524Mi/s, 13h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.614 MiB/s, ETA 1w5d14h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m50.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.524Mi/s, 13h
2|run2 | 2025/02/12 20:48:01 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 40 (retry=false): Post "https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0059": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0059", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.138 MiB/s, ETA 1w6d10h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m50.9s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:01 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 40: Post "https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0059": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.138 MiB/s, ETA 1w6d10h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m50.9s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.138 MiB/s, ETA 1w6d10h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:01 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 190 (retry=false): Post "https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0022": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0022", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.138 MiB/s, ETA 1w6d10h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.2s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:01 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 190: Post "https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0022": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.138 MiB/s, ETA 1w6d10h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.2s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.138 MiB/s, ETA 1w6d10h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:01 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: multi-thread copy: chunk 80/11650 failed: multi-thread copy: failed to write chunk: Post "https://pod-020-3011-16.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0031": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.138 MiB/s, ETA 1w6d10h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:01 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 17 (retry=false): Post "https://pod-020-3011-03.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0029": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-03.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0029", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.138 MiB/s, ETA 1w6d10h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:01 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 17: Post "https://pod-020-3011-03.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0029": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 7.138 MiB/s, ETA 1w6d10h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Sending chunk 210 length 33554432
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.6s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 210 (retry=false): Post "https://pod-020-3011-09.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0054": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-09.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0054", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.6s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 210: Post "https://pod-020-3011-09.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0054": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.6s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 143 (retry=false): Post "https://pod-020-3011-19.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0007": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-19.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0007", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.7s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 143: Post "https://pod-020-3011-19.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0007": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m51.7s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 7.054Mi/s, 14h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: multi-thread copy: chunk 183/11650 failed: multi-thread copy: failed to write chunk: Post "https://pod-020-3011-06.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0010": context canceled
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: multi-thread copy: chunk 210/11650 failed: multi-thread copy: failed to write chunk: Post "https://pod-020-3011-13.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0013": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: multi-thread copy: chunk 212/11650 failed: multi-thread copy: failed to write chunk: Post "https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0028": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 155 (retry=false): Post "https://pod-020-3011-11.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0038": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-11.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0038", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:02 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 155: Post "https://pod-020-3011-11.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0038": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.5s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:03 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: multi-thread copy: chunk 125/11650 failed: multi-thread copy: failed to write chunk: Post "https://pod-020-3011-15.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0053": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.6s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:03 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Sending chunk 205 length 33554432
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.6s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:03 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 205 (retry=false): Post "https://pod-020-3011-07.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0037": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-07.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0037", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.6s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:03 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 205: Post "https://pod-020-3011-07.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0037": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.6s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:03 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 89 (retry=false): Post "https://pod-020-3011-03.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0024": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-03.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0024", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.6s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:03 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 89: Post "https://pod-020-3011-03.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0024": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.692 MiB/s, ETA 2w7h42m
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.6s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | 2025/02/12 20:48:03 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: multi-thread copy: chunk 186/11650 failed: multi-thread copy: failed to write chunk: Post "https://pod-020-3011-16.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0008": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.273 MiB/s, ETA 2w1d6h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m52.7s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.613Mi/s, 15h
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.273 MiB/s, ETA 2w1d6h
2|run2 | Checks: 3 / 3, 100%, Listed 312
2|run2 | Transferred: 0 / 22, 0%
2|run2 | Elapsed time: 3m53.0s
2|run2 | Transferring:
2|run2 | * zlib2-data/pilimi-zlib2-15000000-15679999.tar: 1% /364.036Gi, 6.200Mi/s, 16h
2|run2 | 2025/02/12 20:48:03 DEBUG : zlib2-data/pilimi-zlib2-15000000-15679999.tar: Error sending chunk 157 (retry=false): Post "https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0006": multi-thread copy: failed to write chunk: invalid character '<' looking for beginning of value: &url.Error{Op:"Post", URL:"https://pod-020-3011-01.backblaze.com/b2api/v1/b2_upload_part/4_zb87219cbda6098ab96340e17_f228491889785e70a_d20250212_m204420_c002_v0203011_t0019_u01739393060115/0006", Err:(*fmt.wrapError)(0xc007674740)}
2|run2 | Transferred: 4.402 GiB / 7.901 TiB, 0%, 6.273 MiB/s, ETA 2w1d6h
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked PR 8384 and the continuation referenced from issue 8315, then trace the low-level retry predicate handling the Backblaze B2 upload-part error shown in the logs. Done means recoverable upload-part failures are retried instead of aborting the multi-threaded upload, with the relevant behavior covered by tests if the PR identifies them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100