cloudflare / cloudflare/workers-sdk

Worker asset upload sometimes silently fails

Open
#15,060 9 comments 0 reactions 0 assignees Claimed by @petebacondarwin View on GitHub
feature:workers-assets package:deploy-helpers
Dominant language
TypeScript
Stars
4.5k
Forks
1.5k
Avg merge
3d 8h
Merged PRs (30d)
187

Description

### What versions & operating system are you using?

Seen in the cf worker build pipeline:
- wrangler ^4.115.0 or ^4.119.0
- @cloudflare/workers-types: ^5.20260804.1
- pnpm/10.1.0 npm/? node/v24.18.0 linux x64

### Please provide a link to a minimal reproduction

https://github.com/DonationWatch/donationwatch

### Describe the Bug

Preface: it seems to only happen with lots of opennext asset cache files.

Randomly for like the last 7 days the worker build step silently fails or fails with a timeout for asset uploads:

- case of timeout being correctly propagated to the build agent:

Image

```
14:45:25.448 Uploaded 9954 of 11804 assets
14:45:32.242 Uploaded 10085 of 11804 assets
15:00:19.719 Build took too long and was timed out
```

- case of timeout not being propagated to the build agent:

Image

```
12:25:44.032 Uploaded 9585 of 11805 assets
12:25:44.355 Uploaded 9719 of 11805 assets
12:25:46.398 ▲ [WARNING] Duplicate key "options" in object literal [duplicate-object-key]
12:25:46.398
12:25:46.398 .open-next/server-functions/default/handler.mjs:259:135964:
12:25:46.398 259 │ ...nmentOffset:F3}},reset:E3}},options:[x2,y2]}),{name:"transform...
12:25:46.398 ╵ ~~~~~~~
12:25:46.398
12:25:46.398 The original key "options" is here:
12:25:46.398
12:25:46.398 .open-next/server-functions/default/handler.mjs:259:134940:
12:25:46.398 259 │ ...ing"}),y2=[V],{name:"arrow",options:x2,async fn(a4){let{x:b2,y...
12:25:46.398 ╵ ~~~~~~~
12:25:46.398
12:25:46.398
12:25:46.398 (node:1639) [DEP0190] DeprecationWarning: Passing args to a child process with shell option true can lead to security vulnerabilities, as the arguments are not escaped, only concatenated.
12:25:46.398 (Use `node --trace-deprecation ...` to show where the warning was created)
12:25:46.595 Success: Deploy command completed
12:25:46.771 ✨ Success! Build completed.
```

I would expect that the second build would also fail but it seems to just silently report as successful.

Note I'm not sure if you can reproduce it with the fake data build of the app (`data:fake`) but it seems to be that around ~10k assets being uploaded it times out.
When trying to reduce the amount of static routes it seems to run through just fine again:

```
16:29:28.424 Uploaded 5913 of 5913 assets
16:29:28.426 ✨ Success! Uploaded 5913 files (6307 already uploaded) (77.29 sec)
16:29:28.426
16:29:29.039 Total Upload: 19260.47 KiB / gzip: 2549.69 KiB
```

Last time it worked before that there were around 14k assets:

```
18:42:00.520 Uploaded 14318 of 14410 assets
18:42:01.563 Uploaded 14410 of 14410 assets
18:42:01.580 ✨ Success! Uploaded 14410 files (6001 already uploaded) (344.69 sec)
18:42:01.580
18:42:02.092 Total Upload: 31063.32 KiB / gzip: 2786.67 KiB
18:42:07.253 Worker Startup Time: 40 ms
```

### Please provide any relevant error logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the DonationWatch minimal reproduction and compare the asset-upload logs for builds near 10,000 files, especially the timeout that is not propagated. Done means a timed-out asset upload reliably reports failure to the build agent instead of completing successfully; pull request #15621 indicates work is already underway.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.