cloudflare / cloudflare/actors

proposal: Core performance features (WAL SQL optimization, PuppeteerScraper, ParallelChunker) & Vitest barrel-import resolution

Open
#124 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
422
Forks
28
PR merge metrics
No merged PRs in 30d

Description

Image

https://github.com/user-attachments/assets/39493e16-4161-413e-a6c6-da2ae958a3f5

Orchestrator DO query finished
Here is the draft for the new GitHub issue, detailing both the new features and the Vite circular dependency fix:

Title: proposal: Core performance features (WAL SQL optimization, PuppeteerScraper, ParallelChunker) & Vitest barrel-import resolution

Description:

markdown

### Proposing Core Performance & Scrape Utilities
We want to propose three utility enhancements for the `@cloudflare/actors` core:
1. **SQL Performance Optimizations**: Explicitly initialize SQLite Durable Object storage with:
```typescript
this.storage.sql.exec('PRAGMA journal_mode=WAL;');
this.storage.sql.exec('PRAGMA synchronous=NORMAL;');
This dramatically improves concurrent read/write throughput and prevents lock contention during heavy actor operations. 2. PuppeteerScraper: A base Actor utilizing Cloudflare Browser Rendering to navigate, hydrated-SPA render, and scrape page content, featuring a robust runtime fallback to a standard fetch call if the binding/service is unavailable. 3. ParallelChunker: A utility actor designed to partition large datasets or batches into size-bounded sub-payloads and execute concurrent mapping operations safely within worker concurrency ceilings.

Fixing a Circular Dependency Bug in Vite/Vitest
While testing these core subclasses, we encountered an evaluation-order issue.

The Bug: Subclasses like PuppeteerScraper.ts and ParallelChunker.ts import the base class Actor from index.ts. However, index.ts (acting as the barrel) imports and re-exports those subclasses. When Vitest runs, Vite resolves index.ts which triggers partial evaluation of the subclasses before Actor is fully defined, throwing: TypeError: Class extends value undefined is not a constructor or null

The Fix: We restructured the package directory:

Extracted the abstract base class Actor and its supporting types to actor.ts.
Made index.ts a pure barrel file (only containing re-exports like export * from "./actor"), eliminating circular initialization paths during builds and tests.

**--maybe worth noting, the vitest issue, and pretty much all the suggested changes, were harvested from my 'OrchestratorDO', running as always, lightning fast on cloudflares edge**

Image

Image

Image

Contributor guide

Open the contributing guide

Research direction

Read the proposed entry points and files: index.ts, actor.ts, PuppeteerScraper.ts, and ParallelChunker.ts. Run the existing Vitest suite to reproduce the barrel-import failure, then clarify which of the three performance and scraping utilities is in scope. Done would require an agreed scope, implementation details, and tests for the selected change.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite, typescript
Domain
backend, database, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.