Large dataclips can still cause runs to be lost
@josephjclark is already working on this.
Since Aug 25, 2026.
- Dominant language
- TypeScript
- Stars
- 21
- Forks
- 23
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 17
Description
Despite a lot of improvements to memory handling and payload processing, I think we're still losing lost runs with large dataclips.
I'm looking at impact water which is regularly building 9.8mb dataclips and lost runs.
I'm not sure sure exactly where the error lies - but I think when processing this 10mb dataclip to send to lightning, either the connection is actually timing out or some process (lightning side or worker side) is exceeded a memory limit and giving in.
The worst thing is that this is dying silently and causing the websocket to disconnect, resulting in a lost run
A few things jump to mind:
- Claude points out the state objects are double encoded in to JSOn strings. bloating the size A 9mb dataclip in memory might acutally serialize to 10mb or more in the event sent to lightning because of the nested JSON escape characters
- We are not getting good sentry reports when these events timeout - that needs fixing
This is the underlying cause behind https://github.com/OpenFn/kit/issues/1072 and https://github.com/OpenFn/kit/issues/1179. Both those issues describe symptoms of this one underlying thing. They both describe a case where an event fails to send and then bad things happen. This issue is the reason teh events fail.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.