OpenFn / OpenFn/kit

Large dataclips can still cause runs to be lost

Open
#1,504 1 comment 0 reactions 1 assignee View on GitHub

@josephjclark is already working on this.

Since Aug 25, 2026.

Dominant language
TypeScript
Stars
21
Forks
23
Avg merge
1d 22h
Merged PRs (30d)
17

Description

Despite a lot of improvements to memory handling and payload processing, I think we're still losing lost runs with large dataclips.

I'm looking at impact water which is regularly building 9.8mb dataclips and lost runs.

I'm not sure sure exactly where the error lies - but I think when processing this 10mb dataclip to send to lightning, either the connection is actually timing out or some process (lightning side or worker side) is exceeded a memory limit and giving in.

The worst thing is that this is dying silently and causing the websocket to disconnect, resulting in a lost run

A few things jump to mind:

  • Claude points out the state objects are double encoded in to JSOn strings. bloating the size A 9mb dataclip in memory might acutally serialize to 10mb or more in the event sent to lightning because of the nested JSON escape characters
  • We are not getting good sentry reports when these events timeout - that needs fixing

This is the underlying cause behind https://github.com/OpenFn/kit/issues/1072 and https://github.com/OpenFn/kit/issues/1179. Both those issues describe symptoms of this one underlying thing. They both describe a case where an event fails to send and then bad things happen. This issue is the reason teh events fail.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.