OpenListTeam / OpenListTeam/OpenList
[Feature] Bound Aliyun Drive active callback downloads and preserve retry semantics
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 24.7k
- Forks
- 2.3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 36
Description
Please confirm the following
- I have read and agree to AGPL-3.0 Section 15. The program is provided "as is" without warranties, and I bear the risks of using it.
- I have read and agree to AGPL-3.0 Section 16.
- I confirm that this description is clear, polite, and intended to help maintainers locate the issue.
- I have read the OpenList documentation.
- I searched existing issues and discussions. #2796 concerns S3 redirect/proxy semantics; closed PR #2375 concerns per-IP limits for
/pand/ap. Neither owns Aliyun callback response-body lifetime. - I believe active callback admission must be handled by OpenList because the AliyundriveOpen driver owns the provider identity and creates the callback URL.
- I confirm that active response-body admission and resumable proxy reads are not currently implemented.
- I believe this has general value for OpenList clients that proxy concurrent range reads through AliyundriveOpen.
- I have not read these checkboxes and therefore ticked all of them; please close this issue.
Feature Description
Please distinguish Aliyun Drive API request-rate limiting from the lifetime of
active callback downloads.
At OpenList commit 1a6cabf45aecf66c6d2ff6c32aed39d50264f43c,
drivers/aliyundrive_open/limiter.go limits the start rate of list, link, and
other API calls per user/application. AliyundriveOpen.Link, however, returns a
plain callback URL. The generic proxy opens and streams that URL after the
driver call has returned, so the driver has no resource representing an active
callback response body.
With concurrent ranged object reads this can produce the provider response:
RequestDeniedByCallback: Callback deny this request reason: ExceedMaxConcurrency
The error is currently surfaced through proxy/S3 paths as an HTTP 500
InternalError. Clients then cannot distinguish capacity pressure from a
permanent object or repository failure. Limiting only link-generation QPS does
not solve this because a callback body can remain active long after the link API
request completes.
This is separate from:
- #2796, which defines direct-link 302 versus proxy compatibility;
- #2968, which bounds recursive S3 listing work;
- closed PR #2375, which limited concurrent
/pand/aprequests per client
IP but did not cover the independent S3 server or share capacity by provider
identity.
The same provider error was also reported historically in AlistGo/alist#7258.
The numeric provider limit should be treated as runtime/configuration policy,
not hard-coded from field reports.
Suggested Solution
Implement this in two reviewable stages.
Stage 1: active callback admission
- Add an active-transfer gate owned by AliyundriveOpen and keyed by the Aliyun
user/application identity, so mounts sharing the same provider capacity also
share admission. - Keep the existing API QPS limiter; it protects a different resource.
- In proxy mode, acquire a permit immediately before the callback HTTP GET and
retain it until response EOF, bodyClose, or context cancellation. - Release exactly once on every success, error, early-close, and cancellation
path. - Make the wait context-cancellable and bounded. Start with a conservative
configurable limit rather than assuming an undocumented provider constant. - Recognize
RequestDeniedByCallback/ExceedMaxConcurrencyas temporary
capacity pressure and retry with bounded jittered backoff. - Preserve the temporary classification at protocol boundaries. For S3, map
persistent capacity pressure to HTTP 503 /SlowDown, not HTTP 500
InternalError.
One possible ownership change is for AliyundriveOpen.Link to return a custom
RangeReader in proxy mode, because that keeps callback acquisition and body
lifetime inside the driver. Redirect mode can continue returning the URL. The
exact API is open for maintainer guidance.
Stage 2: stalled-body recovery
After admission is measured and stable:
- Detect inactivity rather than imposing one fixed total-file timeout.
- Close the stalled body first, thereby releasing its permit.
- Reacquire a valid callback URL and resume with
Rangefrom the last byte
successfully delivered. - Validate file identity/version, total size, and
Content-Rangebefore
continuing, so retries cannot duplicate or omit bytes. - Bound resume attempts and use jittered backoff.
Please do not use speculative duplicate/hedged downloads before admission is in
place; each hedge consumes another scarce callback slot.
Validation and observability
Suggested deterministic tests:
- permit held for the complete body lifetime;
- EOF, early
Close, cancellation, callback rejection, and read failure each
release exactly once; - several mounts with the same provider identity share a limit;
- unrelated accounts/providers are isolated;
- queue cancellation does not start a callback request;
- resumed ranges are contiguous with no duplicate or missing bytes;
- S3 overload response remains retryable and is encoded as
SlowDown/503.
Useful metrics/log fields are active count, queue time, time to first byte,
bytes transferred, resume count, provider error code, and terminal reason.
Signed URLs and credentials must not be logged.
Additional Information
This proposal does not remove direct-link optimization, change the S3 listing
work in #2968, or impose a global OpenList connection limit. Its goal is to make
native proxy mode preserve the provider's actual capacity and failure
semantics.
References:
- https://github.com/OpenListTeam/OpenList/issues/2796
- https://github.com/OpenListTeam/OpenList/pull/2968
- https://github.com/OpenListTeam/OpenList/pull/2375
- https://github.com/AlistGo/alist/issues/7258
- Complementary direct-read client proposal:
https://github.com/rustic-rs/rustic_core/issues/555
AI Generated Content
OpenAI Codex assisted with source tracing, architecture analysis, test design,
and wording. The submission was reviewed against the cited OpenList source
revision, observed provider error, contribution guide, and issue template.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with drivers/aliyundrive_open/limiter.go and AliyundriveOpen.Link, then trace the generic proxy callback GET and the S3 error-mapping paths. Review the issue's deterministic validation cases for admission, release, cancellation, shared provider identity, retries, and resumable ranges. Done means the design is agreed with maintainers and those behaviors are covered without changing direct-link mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100