get-convex / get-convex/agent

Exact-stream delta cleanup can exceed the transaction read limit on valid rows

Open
#358 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
349
Forks
92
Avg merge
3d 10h
Merged PRs (30d)
24

Description

On published `@convex-dev/agent@0.7.1`, `streams.deleteStreamAsync` reads up to 1,000 delta documents through a helper paginator without a byte bound. Thirty individually valid 600 KB delta documents exceed the 16 MiB transaction read limit, so cleanup fails before those documents are removed.

## Reproduction

Use a fresh directory with Node 22 and these exact dependencies (no app code, deployment or provider credentials):

```bash
npm init -y
npm install --save-exact @convex-dev/agent@0.7.1 convex@1.45.0 convex-test@0.0.56 convex-helpers@0.1.109 ai@7.0.93 @ai-sdk/provider@4.0.10 @ai-sdk/provider-utils@5.0.36 zod@3.25.76 react@19.2.3 vitest@3.2.4
npx vitest run --config vitest.config.mjs
```

`vitest.config.mjs`:

```js
export default { test: { environment: 'node', include: ['repro.test.ts'], fileParallelism: false, maxWorkers: 1, testTimeout: 30000 } };
```

`repro.test.ts`:

```ts
import { afterEach, describe, expect, it, vi } from 'vitest';
import { convexTest } from 'convex-test';
import componentTest from '@convex-dev/agent/test';
import { componentsGeneric, defineSchema } from 'convex/server';
import { createThread, saveMessage, type AgentComponent } from '@convex-dev/agent';

const component = componentsGeneric().agent as unknown as AgentComponent;
function setup() {
const t = convexTest({
schema: defineSchema({}),
modules: { './_generated/server.ts': async () => ({}) },
transactionLimits: true,
});
componentTest.register(t);
return t;
}
afterEach(() => vi.useRealTimers());

it('deletes individually legal large delta documents through byte-bounded continuations', async () => {
vi.useFakeTimers();
const t = setup();
const threadId = await t.mutation(ctx => createThread(ctx, component));
const streamId = await t.mutation(component.streams.create, { threadId, order: 0, stepOrder: 0, format: 'UIMessageChunk' });
for (let i = 0; i < 30; i++) await t.mutation(component.streams.addDelta, {
streamId, start: i, end: i + 1, parts: [{ type: 'text-delta', id: 't', delta: 'x'.repeat(600_000) }],
});
await t.mutation(component.streams.deleteStreamAsync, { streamId });
await t.finishAllScheduledFunctions(vi.runAllTimers);
expect(await t.query(component.streams.list, { threadId, statuses: ['streaming', 'finished', 'aborted'] })).toEqual([]);
expect(await t.query(component.streams.listDeltas, { threadId, cursors: [{ streamId, cursor: 0 }] })).toEqual([]);
});
```

## Observed and expected

The pristine test rejects with `Read too much data in a single function execution (limit: 16777216 bytes)`. Expected: all stream/delta records are removed through bounded continuations.

Reproduced against pristine 0.7.1 source files whose SHA-256 values were checked against the original package. `convex-test` transaction limits are enabled.

## Suggested fix

Use native indexed pagination with both a document ceiling and `maximumBytesRead`, preserving durable cursors. Our local source/distribution patch uses 64 documents and 1 MiB per page and makes the regression pass. These are defensive application limits, not a proposed universal component cap.

Contributor guide

Open the contributing guide

Research direction

Start by running repro.test.ts with the provided vitest.config.mjs and transaction limits enabled. Inspect streams.deleteStreamAsync and its helper paginator, then apply the suggested indexed pagination with durable cursors and document and byte bounds. Done means the regression passes and all stream and delta records are removed without exceeding the transaction read limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, databases, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.