Project Refactoring Proposal: lance-graph

Current Status: Beta (v0.3.x)
Target: Production-Grade GraphRAG Engine (v1.0)
Core Objective: Transition from a "Cypher-to-SQL Transpiler" to a "Native Neuro-Symbolic Query Engine" capable of handling both semantic vectors and graph topology in a single execution pass.

1. Architectural Diagnosis

The current architecture of lance-graph tightly couples the Cypher Parser (nom), the Logical Planner, and the DataFusion Execution Engine into a single crate. This creates several friction points:

Bloated Dependencies: Consumers who just want to parse Cypher or validate schemas must pull in the heavy datafusion and tokio dependencies.
Rigid Execution: The engine translates Cypher directly to DataFusion LogicalPlans. This makes it difficult to inject custom "Graph-Native" operators (like recursive graph traversal or vector search) that don't map cleanly to SQL joins.
Python Friction: The Python bindings are likely second-class citizens, making integration with AI frameworks (LangChain/LlamaIndex) difficult.

2. Structural Refactoring (The Workspace Pattern)

We recommend migrating the repository to a Cargo Workspace structure. This aligns with lance and lance-context, enforcing clean separation of concerns.

Proposed Directory Layout

lance-graph/
├── Cargo.toml            # Workspace definition
├── README.md             # Unified documentation
├── docs/                 # Architecture Decision Records (ADRs)
├── crates/
│   ├── lance-graph/           # The "Facade" Crate (Re-exports everything)
│   │   └── src/lib.rs
│   │
│   ├── lance-graph-core/      # Pure Rust types (No heavy dependencies)
│   │   ├── src/ast/           # Cypher AST definitions
│   │   ├── src/schema.rs      # GraphConfig, NodeMapping, RelationshipMapping
│   │   └── src/error.rs
│   │
│   ├── lance-graph-planner/   # The Compiler (Cypher -> LogicalPlan)
│   │   └── src/translator.rs  # Rewrites AST into DataFusion LogicalPlan
│   │
│   └── lance-graph-physical/  # Custom Execution Kernels
│       ├── src/traversal.rs   # Optimized Variable-Length Path operator
│       └── src/vector.rs      # Custom Vector Search UDFs
│
├── python/               # First-class Python Package
│   ├── Cargo.toml        # PyO3 bindings
│   ├── pyproject.toml    # Maturin build config
│   └── lance_graph/      # Python wrapper code
│
└── examples/             # End-to-end GraphRAG examples

3. Technical Roadmap & Next Steps

Phase 1: Decoupling & Stability (The "Core" Fixes)

Goal: Fix existing bugs and stabilize the foundation.

[ ] Extract lance-graph-core: Move GraphConfig and AST definitions into a lightweight crate. This allows other tools (like lance-context) to import graph schemas without importing the query engine.
[ ] Fix Column Naming (Issue #30): The current transpiler relies on implicit DataFusion naming, which breaks when DataFusion optimizes plans.
- Action: Implement a strict "Final Projection" pass that forces output column names to match the user's Cypher aliases explicitly.
[ ] Implement UNWIND (Issue #75): This is critical for RAG.
- Action: Map Cypher UNWIND to DataFusion's unnest operator.

Phase 2: The GraphRAG Capabilities

Goal: Enable hybrid search (Vector + Graph) within a single query.

[ ] Native Vector Search Operator: Currently, users must do vector search outside lance-graph and pass IDs in.
- Action: Implement a Cypher custom function vector_search(node.embedding, $query_vector).
- Optimization: Push this down to the Lance Scanner so it uses the IVF-PQ index instead of scanning all nodes.
[ ] Variable Length Paths: Optimize MATCH (a)-[*1..3]->(b).
- Current State: Likely compiles to a fixed chain of JOINs (slow).
- Target: Implement a custom DataFusion UserDefinedLogicalNode for recursive traversal that utilizes Lance's random-access capabilities.

Phase 3: Mutability (The "Delta" Layer)

Goal: Allow agents to write to the graph (add nodes/edges).

Since Lance is immutable, we cannot support direct CREATE or SET easily.

[ ] Design "Graph Delta":
- Create a temporary DeltaTable (Arrow IPC file) for new nodes/edges created during a session.
- Update the Planner to inject a UNION ALL that reads from both the Base Lance Table and the Delta Table.
- Implement compact() to merge Deltas into Lance files periodically.

Phase 4: Python Ecosystem Integration

Goal: Make it the default GraphRAG engine for Python agents.

[ ] Async PyO3 Bindings: Ensure query() releases the Python GIL so agents can run parallel queries.
[ ] LangChain Adapter: Build a LanceGraphRetriever for LangChain that accepts natural language, converts to Cypher (via LLM), and executes.
[ ] NetworkX Export: Add a to_networkx() method to result sets for easy visualization in Python.

Project Refactoring Proposal: lance-graph

Description