cloudflare / cloudflare/agents
@cloudflare/shell: support custom SQL storage backends for large multi-tenant workspaces
- Dominant language
- TypeScript
- Stars
- 5.6k
- Forks
- 711
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 53
Description
## Context
I am evaluating `@cloudflare/shell` for a large multi-tenant web application where each user can own multiple workspaces, and each workspace may contain a large number of file objects.
From the current API shape, `Workspace` can be backed by a SQL-like storage object, for example `this.ctx.storage.sql` inside an Agent / Durable Object, or a D1-compatible SQL backend. This is convenient for small and medium workspaces, but it raises a storage scalability concern for large multi-tenant products.
## Problem
Even if we avoid a single shared D1 database and use `this.ctx.storage.sql` per Agent / Durable Object, the storage pressure only moves from "one global D1 database can grow too large" to "one workspace / Agent / Durable Object can grow too large".
For my target scenario:
- one user can have many workspaces
- one workspace can contain many file objects
- file metadata, directory indexes, versions, snapshots, operation state, and temporary shell state may all grow over time
- some workspaces may represent real code repositories with thousands or tens of thousands of files
- long-lived multi-tenant usage makes quota, compaction, migration, and sharding important
This makes it risky to rely only on D1 or Durable Object SQLite as the metadata store for `@cloudflare/shell` workspaces. R2 is a good fit for large blobs, but the metadata/index layer can still become large and hot.
## Feature request
Would it be possible to make the SQL storage layer in `@cloudflare/shell` more explicitly pluggable, with first-class support for custom `storage.sql`-like objects and database types beyond D1 / Durable Object SQLite?
In particular, I would like to understand whether `@cloudflare/shell` could support a backend interface that can be implemented on top of databases such as PostgreSQL, especially PostgreSQL deployments that are reachable from Workers through an edge-friendly connection pool / Hyperdrive-like setup.
A possible direction could be:
- define a documented SQL backend contract for `@cloudflare/shell`
- expose a `databaseType` or dialect capability flag, for example `sqlite`, `d1`, `durable-object-sqlite`, `postgres`
- keep the default implementation optimized for SQLite/D1
- allow custom adapters to translate the required operations into PostgreSQL-compatible SQL
- document which SQL features are required by `@cloudflare/shell`
- document the expected consistency, transaction, locking, and pagination behavior
- make it possible for large deployments to move metadata/index storage to a horizontally scalable external database while still using R2 for large file blobs
## Why this matters
For small workspaces, the current storage model is simple and attractive. For large multi-tenant products, however, the storage limits and single-object/single-database growth pattern become a core architectural risk.
A custom SQL backend would let applications choose the right storage topology:
- D1 for small/global control-plane metadata
- Durable Object SQLite for hot per-agent state
- R2 for large file contents and snapshots
- PostgreSQL for large workspace metadata, indexes, search-friendly structures, quota accounting, and cross-workspace queries
## Questions
1. Is `@cloudflare/shell` intended to support large, long-lived, multi-tenant workspace storage, or is it primarily designed as a local runtime workspace/cache layer for Agents?
2. Is the current SQL backend interface considered public/stable enough for custom adapters?
3. Would the maintainers be open to a documented custom SQL storage interface with database dialect support?
4. Are there assumptions in `@cloudflare/shell` that would make PostgreSQL support difficult, such as SQLite-specific statements, transaction semantics, or local-only locking behavior?
5. If direct PostgreSQL support is out of scope, would maintainers accept documentation clarifying the intended storage limits and recommended R2 / metadata-sharding architecture for large workspaces?
## Suggested outcome
At minimum, it would be helpful to have documentation that clarifies the intended storage model and scalability boundaries of `@cloudflare/shell`.
Ideally, `@cloudflare/shell` would expose a stable storage adapter interface so applications can implement a PostgreSQL-backed metadata store while continuing to use the package's workspace abstractions.
Contributor guide
Assessment
This issue has not been assessed yet.