Exact-stream delta cleanup can exceed the transaction read limit on valid rows
- Dominant language
- TypeScript
- Stars
- 349
- Forks
- 92
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 24
Description
On published `@convex-dev/agent@0.7.1`, `streams.deleteStreamAsync` reads up to 1,000 delta documents through a helper paginator without a byte bound. Thirty individually valid 600 KB delta documents exceed the 16 MiB transaction read limit, so cleanup fails before those documents are removed.
## Reproduction
Use a fresh directory with Node 22 and these exact dependencies (no app code, deployment or provider credentials):
```bash
npm init -y
npm install --save-exact @convex-dev/agent@0.7.1 convex@1.45.0 convex-test@0.0.56 convex-helpers@0.1.109 ai@7.0.93 @ai-sdk/provider@4.0.10 @ai-sdk/provider-utils@5.0.36 zod@3.25.76 react@19.2.3 vitest@3.2.4
npx vitest run --config vitest.config.mjs
```
`vitest.config.mjs`:
```js
export default { test: { environment: 'node', include: ['repro.test.ts'], fileParallelism: false, maxWorkers: 1, testTimeout: 30000 } };
```
`repro.test.ts`:
```ts
import { afterEach, describe, expect, it, vi } from 'vitest';
import { convexTest } from 'convex-test';
import componentTest from '@convex-dev/agent/test';
import { componentsGeneric, defineSchema } from 'convex/server';
import { createThread, saveMessage, type AgentComponent } from '@convex-dev/agent';
const component = componentsGeneric().agent as unknown as AgentComponent;
function setup() {
const t = convexTest({
schema: defineSchema({}),
modules: { './_generated/server.ts': async () => ({}) },
transactionLimits: true,
});
componentTest.register(t);
return t;
}
afterEach(() => vi.useRealTimers());
it('deletes individually legal large delta documents through byte-bounded continuations', async () => {
vi.useFakeTimers();
const t = setup();
const threadId = await t.mutation(ctx => createThread(ctx, component));
const streamId = await t.mutation(component.streams.create, { threadId, order: 0, stepOrder: 0, format: 'UIMessageChunk' });
for (let i = 0; i < 30; i++) await t.mutation(component.streams.addDelta, {
streamId, start: i, end: i + 1, parts: [{ type: 'text-delta', id: 't', delta: 'x'.repeat(600_000) }],
});
await t.mutation(component.streams.deleteStreamAsync, { streamId });
await t.finishAllScheduledFunctions(vi.runAllTimers);
expect(await t.query(component.streams.list, { threadId, statuses: ['streaming', 'finished', 'aborted'] })).toEqual([]);
expect(await t.query(component.streams.listDeltas, { threadId, cursors: [{ streamId, cursor: 0 }] })).toEqual([]);
});
```
## Observed and expected
The pristine test rejects with `Read too much data in a single function execution (limit: 16777216 bytes)`. Expected: all stream/delta records are removed through bounded continuations.
Reproduced against pristine 0.7.1 source files whose SHA-256 values were checked against the original package. `convex-test` transaction limits are enabled.
## Suggested fix
Use native indexed pagination with both a document ceiling and `maximumBytesRead`, preserving durable cursors. Our local source/distribution patch uses 64 documents and 1 MiB per page and makes the regression pass. These are defensive application limits, not a proposed universal component cap.
Contributor guide
Research direction
Start by running repro.test.ts with the provided vitest.config.mjs and transaction limits enabled. Inspect streams.deleteStreamAsync and its helper paginator, then apply the suggested indexed pagination with durable cursors and document and byte bounds. Done means the regression passes and all stream and delta records are removed without exceeding the transaction read limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100