[RFC] AgentDB v2.0: Multi-Database ORM with Universal Vector Support

🎯 Vision

Transform AgentDB from a SQLite-only vector database into a universal ORM-based vector database supporting PostgreSQL, MySQL, MongoDB, and specialized vector databases, while maintaining 100% backward compatibility.

---

📋 Overview

This proposal outlines the implementation of a multi-database ORM layer for AgentDB v2.0, enabling users to seamlessly switch between databases via configuration without code changes.

Key Principles:

• ✅ Zero Breaking Changes - All v1.x code continues to work
• ✅ Configuration-Based - Switch databases via environment variables
• ✅ Native Vector Support - Leverage each database's vector capabilities
• ✅ Type-Safe - Full TypeScript support with ORM pattern

---

📊 Supported Databases (v2.0 Roadmap) - UPDATED ⚡

Database	Phase	Priority	Vector Support	Best For
SQLite	✅ Existing	Baseline	Custom HNSW	Embedded, Edge, Development
Neon	🎯 Phase 1	🥇 PRIMARY	pgvector (HNSW/IVFFlat)	Serverless, Modern Apps, CI/CD
PostgreSQL	🎯 Phase 1	🥈 Secondary	pgvector (HNSW/IVFFlat)	Enterprise, On-Premises
Supabase	📅 Phase 3	-	pgvector	Full-Stack Applications
Turso	📅 Phase 3	-	Custom	Edge Computing
MySQL	📅 Phase 3	-	Custom	Traditional Apps
MongoDB	📅 Phase 3	-	Native	Document-Heavy
Pinecone	📅 Phase 4	-	Native	Billion-Scale
Qdrant	📅 Phase 4	-	Native	Vector-First

🚀 Why Neon is Phase 1 Primary

Key Advantages:

1. ⚡ Instant Database Branching - <1s, perfect for CI/CD
2. 💰 70-90% Cost Savings - Free tier, scale-to-zero
3. ⏱️ 3-Minute Setup - vs 8 hours self-hosted (160x faster)
4. 🔥 Sub-Second Cold Starts - 500ms P50
5. 🔓 Zero Lock-In - Pure PostgreSQL, easy migration

See comment below for detailed analysis and implementation strategy.

---

🚀 Key Features

1. Zero Breaking Changes

```typescript
// Existing v1.x code works unchanged in v2.0
const db = new SQLiteVectorDB({ path: 'vectors.db' });
db.insert({ embedding: [0.1, 0.2, 0.3] });

// New v2.0 API adds database flexibility
const db = new AgentDB({
backend: BackendType.NEON, // or POSTGRES_PGVECTOR
connection: { url: process.env.DATABASE_URL }
});
```

2. Configuration-Based Migration

```typescript
// Same code, different database via env variables
const db = new AgentDB({
backend: process.env.DB_BACKEND as BackendType || BackendType.SQLITE_NATIVE,
connection: { url: process.env.DATABASE_URL },
path: process.env.SQLITE_PATH || './dev.db'
});

// Development: DB_BACKEND=neon DATABASE_URL=postgresql://neon...
// Production: DB_BACKEND=postgres-pgvector DATABASE_URL=postgresql://...
```

3. Native Vector Extensions

• Neon & PostgreSQL: pgvector extension (HNSW + IVFFlat indexes)
• MongoDB: Native `$vectorSearch`
• MySQL: Custom vector ops (future: MySQL 9.0)
• SQLite: Custom HNSW implementation (existing)

4. Type-Safe ORM Query Builder

```typescript
const results = await db.query

()
.similarTo(queryEmbedding, { k: 10, metric: 'cosine' })
.where('metadata.category', '=', 'tech')
.whereBetween('published', '2024-01-01', '2024-12-31')
.orderBySimilarity('desc')
.execute();
```

---

🏗️ Architecture Design

Current Architecture (v1.x)

```
SQLiteVectorDB
├── NativeBackend (better-sqlite3)
└── WasmBackend (sql.js)
```

Proposed Architecture (v2.0)

```
AgentDB
├── Core
│ ├── DatabaseAdapter (abstract base class)
│ ├── QueryBuilder (database-agnostic SQL)
│ ├── ConnectionManager (connection pooling)
│ └── SchemaManager (migrations)
├── Adapters
│ ├── SQLite (NativeAdapter, WasmAdapter)
│ ├── PostgreSQL
│ │ ├── NeonAdapter (PRIMARY - serverless features)
│ │ └── PostgresAdapter (self-hosted)
│ ├── MySQL, MongoDB
│ └── Cloud (Supabase, Turso)
└── ORM
├── Repository Pattern
├── Entity Management
└── Migration System
```

---

📈 Implementation Timeline - UPDATED

Total Duration: ~5.5 months (22 weeks)

Phase 1: PostgreSQL Foundation with Neon Priority (Weeks 1-4)

Primary Goal: Neon serverless PostgreSQL + self-hosted fallback

Week 1-2:

• ✅ Base `DatabaseAdapter` abstraction
• ✅ `PostgresAdapter` (works for both Neon + self-hosted)
• ✅ Query builder for SQL generation
• ✅ Connection pooling

Week 2-3:

• ✅ `NeonAdapter` extending `PostgresAdapter`
• ✅ Database branching API integration
• ✅ Neon-specific optimizations
• ✅ CI/CD integration examples

Week 3-4:

• ✅ Comprehensive test suite
• ✅ Documentation (Neon quickstart, migration guides)
• ✅ Example projects with Neon branching
• ✅ Performance benchmarks

Deliverable: v2.0.0-alpha.1 (Neon + PostgreSQL support)

Phase 2: ORM Layer (Weeks 5-7)

• Repository pattern, Type-safe queries, Migration system
  Deliverable: v2.1.0-alpha

Phase 3: Multi-Database (Weeks 8-13)

• MySQL, MongoDB, Supabase, Turso
  Deliverable: v2.2.0-beta

Phase 4: Vector Databases (Weeks 14-17)

• Pinecone, Weaviate, Qdrant, Milvus
  Deliverable: v2.3.0-beta

Phase 5: Production Hardening (Weeks 18-20)

• Performance, Security, Load Testing
  Deliverable: v2.4.0-rc

Phase 6: Stable Release (Weeks 21-22)

• Documentation, Migration guides, Launch
  Deliverable: v2.0.0 (stable)

---

💡 Use Case Examples

Example 1: Development → Production (Neon-Powered)

```typescript
// Development (Neon free tier)
const devDB = new AgentDB({
backend: BackendType.NEON,
connection: {
url: process.env.NEON_DEV_URL
}
});

// Production (Neon scale plan OR self-hosted)
const prodDB = new AgentDB({
backend: BackendType.NEON, // or POSTGRES_PGVECTOR for self-hosted
connection: {
url: process.env.DATABASE_URL,
pool: { min: 5, max: 20 }
},
vectorExtension: {
pgvector: {
dimensions: 1536,
indexType: 'hnsw'
}
}
});
```

Example 2: CI/CD with Database Branching

```yaml

.github/workflows/test.yml

• name: Create isolated test database
  run: |
  BRANCH_NAME="ci-${{ github.sha }}"
  neon branches create --name $BRANCH_NAME
  echo "DATABASE_URL=$(neon connection-string $BRANCH_NAME)" >> $GITHUB_ENV

• name: Run AgentDB tests
  run: npm test # Uses isolated database branch

• name: Cleanup
  run: neon branches delete ci-${{ github.sha }}
  ```

Example 3: Multi-Tenant SaaS

```typescript
import { createClient } from '@neondatabase/api-client';

async function provisionTenant(tenantId: string) {
// Create isolated database for tenant in <1 second
const branch = await neon.createBranch({
name: `tenant-${tenantId}`,
parentBranch: 'main'
});

return new AgentDB({
backend: BackendType.NEON,
connection: { url: branch.connectionString }
});
}

// Each tenant gets:
// - Isolated database
// - Same schema as main
// - Zero storage cost until they write data
```

---

📊 Performance Targets

Operation	SQLite	Neon	PostgreSQL	Target
Insert (single)	0.5ms	0.8ms	0.8ms	<1ms
Insert (batch 1K)	45ms	60ms	60ms	<100ms
Search (k=10)	2ms	3ms	3ms	<5ms
Search (k=100)	12ms	15ms	15ms	<20ms
Cold Start	N/A	500ms	N/A	<1s

---

🔄 Migration Strategy

Data Migration Tool

```bash

CLI tool for migrating between databases

npx agentdb migrate \
--from sqlite:./vectors.db \
--to neon:postgresql://neon.tech/agentdb \
--batch-size 1000 \
--create-indexes \
--validate

Or migrate from Neon to self-hosted

npx agentdb migrate \
--from neon:postgresql://neon.tech/agentdb \
--to postgres:postgresql://localhost/agentdb
```

Backward Compatibility Timeline

• v2.0-2.4: `SQLiteVectorDB` works (deprecation warning)
• v3.0+: `SQLiteVectorDB` removed, use `AgentDB`

---

📚 Documentation Requirements

User Documentation

1. Neon Quickstart (3-minute setup)
2. Migration Guide - v1.x to v2.0, SQLite to Neon
3. Configuration Guide - All databases
4. API Reference - Complete API docs
5. Database Guides - Neon, PostgreSQL, MySQL, MongoDB

Developer Documentation

1. Architecture Overview
2. Adding New Adapters
3. Testing Guide
4. Performance Optimization

---

🎯 Success Metrics

Technical Metrics

• ✅ Zero breaking changes (100% v1.x compatibility)
• ✅ <10% performance overhead vs native
• ✅ 100% test coverage for all adapters
• ✅ <10ms p95 latency for vector searches
• ✅ 70-90% cost reduction for dev/test (Neon)

Adoption Metrics

• 50% of new users start with Neon (serverless)
• 1,000+ downloads/week within 3 months
• 100+ production deployments within 1 year

---

🚨 Risks & Mitigation

Risk	Impact	Mitigation
Breaking changes	HIGH	Strict backward compatibility testing
Neon vendor lock-in	MEDIUM	Pure PostgreSQL = easy migration
Performance regression	MEDIUM	Continuous benchmarking
Database-specific bugs	MEDIUM	Comprehensive test suite per adapter

---

📖 Related Documents

Comprehensive planning documents in `/docs/plans/`:

1. Executive Summary
2. Full Migration Plan (~70 pages)
3. Quick Reference
4. Type Definitions (~500 lines)
5. Neon Benefits Analysis (NEW!)

---

🤝 Contributing

Areas where you can help:

• Implement Neon/PostgreSQL adapters
• Write tests and benchmarks
• Create documentation and examples
• Provide feedback on API design

---

✅ Next Steps

Immediate (Week 1)

• Community feedback on Neon prioritization
• Prototype Neon adapter
• Set up CI/CD with Neon branching
• Performance benchmarking baseline

Short-term (Weeks 2-4)

• Implement `DatabaseAdapter` base class
• Complete `NeonAdapter` + `PostgresAdapter`
• Write comprehensive test suite
• Release v2.0.0-alpha.1

---

Planning Status: ✅ Complete (Updated with Neon priority)
Implementation Status: 📅 Ready to Start
Timeline: ~5.5 months from approval
Backward Compatibility: 100% guaranteed