Proposal: create an official lance-go project under lance-format

Proposal: create a lance-go repo under lance-format org

Summary

I would like to propose creating an official Go binding for Lance as a standalone repository under the lance-format organization, for example github.com/lance-format/lance-go.

I built a small proof of concept here:

https://github.com/wirybeaver/lance-go-demo

The PoC validates that Go can read Lance datasets through the existing lance-c ABI:

• open a Lance dataset
• read version, schema, and row count metadata
• scan record batches through the Arrow C Data Interface
• support projection, SQL filters, limit, and batch size
• keep the public API read-only for the initial phase
• build against local lance-c source without vendoring native binaries or headers

Why Lance needs Go bindings

Go is common in backend services, data infrastructure, control planes, indexing services, and operational tooling. Today, Go applications that need Lance data generally have to shell out to Python, run sidecar services, or reimplement partial readers. None of those options are ideal for services that need a simple embedded read path.

An official lance-go package would make Lance easier to adopt in Go systems while keeping the implementation aligned with the core Lance project. It would also give users one documented compatibility story instead of several unofficial bindings with different native dependency assumptions.

Proposed approach

Create a standalone lance-go repository under lance-format, backed by the official lance-c ABI.

The Go package should not reimplement Lance. Instead, it should use lance-c as the native compatibility boundary and use Arrow Go's arrow/cdata package to move schemas and record batches across cgo.

How the pieces fit together

At build time, lance-go is a normal Go module plus a cgo bridge. The Go compiler sees the public Go API, cgo sees lance/lance.h, and the final package links to the native lance-c library.

Build time

+-----------------------+        #include <lance/lance.h>        +-----------------------+
| Go application        |  ------------------------------------>  | lance-c headers       |
| imports lance-go      |                                        | LanceDataset, scanner |
+-----------------------+                                        +-----------------------+
            |                                                              |
            | go build / go test                                           |
            v                                                              v
+-----------------------+        cgo compile + native link       +-----------------------+
| lance-go              |  ------------------------------------>  | liblance_c.a / .so    |
| Go API + cgo bridge   |                                        | official C ABI        |
+-----------------------+                                        +-----------------------+
                                                                           |
                                                                           v
                                                               +-----------------------+
                                                               | Lance Rust crates     |
                                                               | dataset/scan/write    |
                                                               +-----------------------+

At runtime, the Go API owns small Go wrapper objects while the actual dataset, scanner, and Arrow stream are native handles owned and released through lance-c.

Runtime sequence for Open + Scan

Go app
  |
  | lance.Open(uri)
  v
lance-go
  |
  | C.lance_dataset_open(uri, storage_opts, version)
  v
lance-c ABI
  |
  | returns *LanceDataset
  v
lance-go Dataset wrapper
  |
  | ds.Scan(WithColumns(...), WithFilter(...), WithLimit(...))
  v
lance-go
  |
  | C.lance_scanner_new(dataset, columns, filter)
  | C.lance_scanner_set_limit(scanner, limit)
  | C.lance_scanner_to_arrow_stream(scanner, out_stream)
  v
lance-c / Lance Rust scanner
  |
  | ArrowArrayStream.get_next(...)
  v
lance-go
  |
  | cdata.ImportCRecordBatchWithSchema(array, schema)
  v
Go caller receives arrow.RecordBatch

A simplified version of the cgo bridge looks like this:

//go:build cgo

package lance

/*
#include <lance/lance.h>
#include <stdlib.h>

static int lance_go_stream_get_next(struct ArrowArrayStream* stream, struct ArrowArray* out) {
    return stream->get_next(stream, out);
}

static int lance_go_array_is_released(struct ArrowArray* array) {
    return array == NULL || array->release == NULL;
}
*/
import "C"

import (
    "io"
    "unsafe"

    "github.com/apache/arrow-go/v18/arrow"
    "github.com/apache/arrow-go/v18/arrow/cdata"
)

func openDataset(uri string) (*C.LanceDataset, error) {
    cURI := C.CString(uri)
    defer C.free(unsafe.Pointer(cURI))

    ds := C.lance_dataset_open(cURI, nil, 0)
    if ds == nil {
        return nil, lastNativeError()
    }
    return ds, nil
}

func scanNext(stream *C.struct_ArrowArrayStream, array *C.struct_ArrowArray, schema *arrow.Schema) (arrow.RecordBatch, error) {
    if status := C.lance_go_stream_get_next(stream, array); status != 0 {
        return nil, streamError(stream, status)
    }
    if C.lance_go_array_is_released(array) != 0 {
        return nil, io.EOF
    }

    return cdata.ImportCRecordBatchWithSchema(
        (*cdata.CArrowArray)(unsafe.Pointer(array)),
        schema,
    )
}

The important point is that cgo is only the language boundary. Query execution, file IO, projection/filter pushdown, scanning, and writing remain in Lance through lance-c.

The initial API can be intentionally small and read-only:

ds, err := lance.Open(uri)
schema, err := ds.Schema()
rows, err := ds.CountRows()

reader, err := ds.Scan(
    lance.WithColumns("id", "vector"),
    lance.WithFilter("id > 100"),
    lance.WithLimit(1000),
)
batch, err := reader.Read()

The PoC uses deterministic cleanup for native handles:

• close LanceDataset
• close LanceScanner
• release ArrowArrayStream
• release/reuse ArrowArray
• set Go finalizers only as a fallback

Proof of concept

Repository:

https://github.com/wirybeaver/lance-go-demo

Smoke test:

make build-native
make example-local

The local smoke test creates a tiny Lance dataset and then reads it from Go. It prints the schema and scanned batch row counts.

Test coverage in the PoC includes:

• opening a dataset and reading dataset version
• counting rows
• importing schema through Arrow C Data
• full scans, projected scans, filtered scans, limited scans
• deterministic close behavior
• native error handling for missing datasets
• CGO_ENABLED=0 fallback behavior

Phased roadmap

Phase 0: minimal read API

• Create the official repository.
• Add source-build instructions for lance-c.
• Expose dataset open, close, version, row count, schema, and scanner APIs.
• Support projection, SQL filter, limit, and batch size.
• Add tests using Go-created Arrow fixtures.

Phase 1: read API hardening

• Add open options for dataset version and storage options.
• Add version history helpers.
• Add fragment IDs and row ID scan support.
• Add random access APIs such as Take.
• Stabilize Go error types around LanceErrorCode.
• Expand examples for local datasets, projected scans, filtered scans, and batch-size tuning.

Phase 2: distribution and CI

• Decide source layout and release process for github.com/lance-format/lance-go.
• Add Linux amd64, Linux arm64, and macOS arm64 CI.
• Support installed lance-c through pkg-config or CMake.
• Keep source builds as the primary supported path first.
• Consider binary downloads only after source builds and release compatibility are stable.

Phase 3: advanced read and search APIs

• Expose vector search.
• Expose full-text search.
• Support Substrait filters.
• Expose index metadata and index segment selection.
• Add conformance tests against Rust/Python-created datasets.
• Add basic scan and search benchmarks.

Phase 4: write and mutation APIs

• Expose create, append, and overwrite from Go Arrow record readers.
• Add delete, update, and merge-insert.
• Add schema evolution helpers.
• Add compaction and index creation/management.
• Keep Arrow ownership rules explicit for buffers crossing cgo.

Phase 5: production readiness

• Publish API docs and examples.
• Define compatibility policy between lance-go and lance-c.
• Add race tests, memory-leak regression tests, and sanitizer-friendly native checks where practical.
• Add cloud-storage integration tests.
• Track any required lance-c ABI gaps upstream.

Productionization discussion

Assuming the right direction is a standalone github.com/lance-format/lance-go repository backed by the lance-c ABI, the main open questions are around production readiness.

ABI gaps for bindings

• Add explicit error-state management, for example lance_clear_last_error, so bindings can safely distinguish successful zero-valued results from stale thread-local errors.
• Expose native version / ABI metadata, for example lance_c_version() and lance_c_abi_version(), so lance-go can validate the linked native library at init time.
• Make ownership contracts explicit for every Arrow C Data path, especially which side releases ArrowSchema, ArrowArray, and ArrowArrayStream on success and failure.
• Prefer status-returning APIs for values where 0 can be valid, for example row counts and version IDs, or add paired out parameters to avoid sentinel ambiguity.

Read performance gaps to discuss

• Confirm that lance_scanner_to_arrow_stream is the preferred high-throughput read path for Go, with projection/filter pushdown happening inside Lance before Arrow batches cross cgo.
• Document batch-size guidance and defaults for Go callers, since cgo overhead should be amortized over record batches rather than per-row calls.
• Consider whether Go needs an explicit prefetch / concurrency control surface for scans, or whether the existing scanner/runtime behavior is already sufficient.
• Expose any useful scan metrics from lance-c, such as bytes read, fragments scanned, batches produced, or filter selectivity, so Go services can tune and observe performance.

Write performance gaps to discuss

• Define the recommended zero-copy write path from Arrow Go record readers through lance_dataset_write.
• Clarify whether Arrow buffers exported from Go must use a C-backed allocator for all write paths, or whether synchronous consumption is sufficient for the current API.
• Expose writer tuning consistently through lance-c, including max rows per file, max rows per group, max bytes per file, data storage version, and stable row IDs.
• Consider append/overwrite/create performance expectations for long-running Go services, including backpressure, batching, and memory ownership during large writes.

Release alignment

• Define how lance-go, lance-c, and core Lance releases should line up.
• One possible model: each lance-go release declares the supported lance-c minor version range, validates the linked ABI at runtime, and follows semver for Go API compatibility.
• Decide whether lance-go should depend on installed lance-c artifacts, build lance-c from source in CI, or eventually offer checksum-verified native binary downloads.
• Decide where release notes should call out compatibility: lance-go changelog, lance-c releases, or both.

Start by reviewing the lance-go-demo proof of concept and running its make build-native and make example-local smoke tests. Then inspect the lance-c ABI and Arrow Go cdata boundary described in the proposal; done would require an agreed repository scope, initial read-only API, source-build path, and test plan.