cloudflare / cloudflare/actors
proposal: Core performance features (WAL SQL optimization, PuppeteerScraper, ParallelChunker) & Vitest barrel-import resolution
- Dominant language
- TypeScript
- Stars
- 422
- Forks
- 28
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/user-attachments/assets/39493e16-4161-413e-a6c6-da2ae958a3f5
Orchestrator DO query finished
Here is the draft for the new GitHub issue, detailing both the new features and the Vite circular dependency fix:
Title: proposal: Core performance features (WAL SQL optimization, PuppeteerScraper, ParallelChunker) & Vitest barrel-import resolution
Description:
markdown
### Proposing Core Performance & Scrape Utilities
We want to propose three utility enhancements for the `@cloudflare/actors` core:
1. **SQL Performance Optimizations**: Explicitly initialize SQLite Durable Object storage with:
```typescript
this.storage.sql.exec('PRAGMA journal_mode=WAL;');
this.storage.sql.exec('PRAGMA synchronous=NORMAL;');
This dramatically improves concurrent read/write throughput and prevents lock contention during heavy actor operations. 2. PuppeteerScraper: A base Actor utilizing Cloudflare Browser Rendering to navigate, hydrated-SPA render, and scrape page content, featuring a robust runtime fallback to a standard fetch call if the binding/service is unavailable. 3. ParallelChunker: A utility actor designed to partition large datasets or batches into size-bounded sub-payloads and execute concurrent mapping operations safely within worker concurrency ceilings.
Fixing a Circular Dependency Bug in Vite/Vitest
While testing these core subclasses, we encountered an evaluation-order issue.
The Bug: Subclasses like PuppeteerScraper.ts and ParallelChunker.ts import the base class Actor from index.ts. However, index.ts (acting as the barrel) imports and re-exports those subclasses. When Vitest runs, Vite resolves index.ts which triggers partial evaluation of the subclasses before Actor is fully defined, throwing: TypeError: Class extends value undefined is not a constructor or null
The Fix: We restructured the package directory:
Extracted the abstract base class Actor and its supporting types to actor.ts.
Made index.ts a pure barrel file (only containing re-exports like export * from "./actor"), eliminating circular initialization paths during builds and tests.
**--maybe worth noting, the vitest issue, and pretty much all the suggested changes, were harvested from my 'OrchestratorDO', running as always, lightning fast on cloudflares edge**
Contributor guide
Research direction
Read the proposed entry points and files: index.ts, actor.ts, PuppeteerScraper.ts, and ParallelChunker.ts. Run the existing Vitest suite to reproduce the barrel-import failure, then clarify which of the three performance and scraping utilities is in scope. Done would require an agreed scope, implementation details, and tests for the selected change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sqlite, typescript
- Domain
- backend, database, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100