Release GIL during remote storage operations to prevent blocking

[Performance] Release GIL during graph execution and storage operations

Context

lance-graph relies on the underlying Lance format for storage and potentially remote object stores (S3, GCS) for persistence. Operations like executing Cypher queries (CypherQuery.execute) or persisting nodes/relationships involve significant I/O.

Problem

If the Python bindings for lance-graph use a "sync-over-async" bridge (e.g., rt.block_on) to call into the Rust core without explicitly releasing the Global Interpreter Lock (GIL), the Python interpreter will freeze during these operations.

This is critical when:

1. Remote Storage is used: Latency for S3/GCS operations (50ms - 500ms+) will block the main thread entirely.
2. Long-running Queries: Complex graph traversals or vector searches running in Rust will starve Python background threads (heartbeats, web server loops, UI) if the GIL is held for the duration of the query.

Proposed Solution

Audit the PyO3 bindings in python/ (likely python/src/lib.rs or similar) and ensure that any blocking runtime execution is wrapped in py.allow_threads.

Implementation Plan

1. Audit & Identification

Locate where Python methods call into async Rust code. Potential targets:

• CypherQuery::execute (if it spawns async tasks for scanning/filtering)
• GraphConfig / Storage initialization
• Any method using tokio::runtime::Runtime::block_on

2. Refactor Bindings

Refactor blocking calls to release the GIL.

Before (Blocking):

#[pyfunction]
fn execute_query(query: String) -> PyResult<PyArrowTable> {
    let rt = Runtime::new().unwrap();
    // Holds GIL while executing graph logic
    rt.block_on(async { engine.execute(&query).await }) 
}

After

#[pyfunction]
fn execute_query(py: Python<'_>, query: String) -> PyResult<PyArrowTable> {
    // 1. Prepare arguments (Clone/Move to Rust types)
    let query_clone = query.clone();

    // 2. Release GIL
    let result = py.allow_threads(move || {
        let rt = Runtime::new().unwrap();
        rt.block_on(async {
            // Expensive graph traversal / IO happens here
            engine.execute(&query_clone).await
        })
    });

    // 3. Handle Result (GIL re-acquired)
    result.map_err(|e| PyValueError::new_err(e.to_string()))
}

3. Verification

Add a concurrency test (python/tests/test_concurrency.py) to verify that a background thread (e.g., a simple counter or heartbeat) continues to run while a heavy graph query is executing.

import threading
import time
from lance_graph import CypherQuery

def test_gil_release_during_query():
    heartbeats = 0
    stop_event = threading.Event()

    def heartbeat():
        nonlocal heartbeats
        while not stop_event.is_set():
            heartbeats += 1
            time.sleep(0.01)

    t = threading.Thread(target=heartbeat)
    t.start()

    try:
        # Run a query that takes some non-trivial time
        # (Mock or use a large enough dataset/complex join)
        query = CypherQuery("MATCH (n) RETURN n")
        query.execute(...)
        
        assert heartbeats > 5, "Background thread was starved! GIL was not released."
    finally:
        stop_event.set()
        t.join()

Acceptance Criteria

• Audit performed on python/ bindings.
• py.allow_threads applied to heavy query/IO paths.
• Concurrency test confirms background thread liveness.

Start by auditing the PyO3 bindings under python/, especially python/src/lib.rs or similar, and locate CypherQuery::execute, storage initialization, and calls to Runtime::block_on. Review how these paths handle Python objects before deciding where py.allow_threads can be applied. Add or update python/tests/test_concurrency.py, and consider the work done when heavy query or I/O paths release the GIL and the background-thread liveness test passes.