Minimal implementation for Dataset::write(...)

Hello,
Thanks for your work on Lance.

I am developing lighweight Rust and Python packages to make data scientists and data engineers to work with lance, for example:

• https://github.com/tuned-org-uk/pygenestore
• https://github.com/tuned-org-uk/genegraph-storage
• https://github.com/tuned-org-uk/javelin-tui

I have started from very basic capabilites, only load/save large numpy arrays and Vec<Vec<f64>>, read and display datasets. Even these simple operations require a compile time from 4 to 8 minutes in the CI pipelines and locally depending on the platform, the libraries are around 40 Mb of size.

My current Cargo.toml looks like:

...
lance = { version = " 0.39.0"}
arrow = "^56.1.0"
arrow-array = "^56.1.0"
...

The first line implies a lot of libraries on which the user has few/no control and some applications may not need all of them.
In particular I couldn't find a way to resolve this import without importing the full lance package:

...
use lance::dataset::{Dataset, WriteMode, WriteParams};
...

    Dataset::write(...)

Link to code

If I could do something:

lance-minimal = { version = " 0.39.0" }  # this includes all the necessary code for a simple store/load roundtrip on a Lance filesystem

and have with that a working minimal implementation that includes "core", "namespace", "io", "dataset" (and params to have a working Lance filesystem), ... that would improve a lot applications development.
Then if a user needs larger packages like datafusion for complex querying and catalogs, it would be easier to provide them in features. For example also having a feature for each cloud provider ("aws", "gcloud", ...).

Please let me know if I missed something, maybe there is a way of creating a dataset without importing the full package.

EDIT:

An example of the sort of trouble that the lack of a minimal package can produce:

cargo check failed to start: Cargo watcher failed, the command produced no valid metadata (exit code: ExitStatus(unix_wait_status(25856))):
    Updating crates.io index
error: failed to select a version for `xz2`.
    ... required by package `datafusion v50.3.0`
    ... which satisfies dependency `datafusion = "^50.0.0"` of package `lance v1.0.0`
    ... which satisfies dependency `lance = "^1.0.0"` of package `genegraph-storage v0.10.0
versions that meet the requirements `^0.1` are: 0.1.7, 0.1.6, 0.1.5, 0.1.4, 0.1.3, 0.1.2, 0.1.1, 0.1.0

package `xz2` links to the native library `lzma`, but it conflicts with a previous package which links to `lzma` as well:
package `liblzma-sys v0.4.3`
    ... which satisfies dependency `liblzma-sys = "^0.4.3"` of package `liblzma v0.4.4`
    ... which satisfies dependency `liblzma = "^0.4.4"` of package `tracel-llvm-bundler v20.1.4-5`
    ... which satisfies dependency `tracel-llvm-bundler = "^20.1.4-5"` of package `cubecl-cpu v0.8.0`
    ... which satisfies dependency `cubecl-cpu = "^0.8.0"` of package `cubecl v0.8.0`
    ... which satisfies dependency `cubecl = "^0.8.0"` of package `burn-wgpu v0.19.0`
    ... which satisfies dependency `burn-wgpu = "^0.19"` 
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the `links = "lzma"` value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

package `datafusion` depends on `xz2` with feature `static` but `xz2` does not have that feature.


failed to select a version for `xz2` which could resolve this conflict

Failed to read Cargo metadata with dependencies for `/datadisk/publish/topolog-embedders/Cargo.toml`: `cargo metadata` exited with an error:     Blocking waiting for file lock on package cache
    Updating crates.io index
error: failed to select a version for `xz2`.
    ... required by package `datafusion v50.3.0`
    ... which satisfies dependency `datafusion = "^50.0.0"` of package `lance v1.0.0`
    ... which satisfies dependency `lance = "^1.0.0"` of package `genegraph-storage v0.10.0`
versions that meet the requirements `^0.1` are: 0.1.7, 0.1.6, 0.1.5, 0.1.4, 0.1.3, 0.1.2, 0.1.1, 0.1.0

package `xz2` links to the native library `lzma`, but it conflicts with a previous package which links to `lzma` as well:
package `liblzma-sys v0.4.3`
    ... which satisfies dependency `liblzma-sys = "^0.4.3"` of package `liblzma v0.4.4`
    ... which satisfies dependency `liblzma = "^0.4.4"` of package `tracel-llvm-bundler v20.1.4-5`
    ... which satisfies dependency `tracel-llvm-bundler = "^20.1.4-5"` of package `cubecl-cpu v0.8.0`
    ... which satisfies dependency `cubecl-cpu = "^0.8.0"` of package `cubecl v0.8.0`
    ... which satisfies dependency `cubecl = "^0.8.0"` of package `burn-wgpu v0.19.0`
    ... which satisfies dependency `burn-wgpu = "^0.19"
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the `links = "lzma"` value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

package `datafusion` depends on `xz2` with feature `static` but `xz2` does not have that feature.


failed to select a version for `xz2` which could resolve this conflict

Start with the dependency declarations in Cargo.toml and the Dataset::write usage in src/traits/lance.rs. Trace which dependencies are required for a basic filesystem round trip and define whether a minimal package or feature split can provide that without the full lance dependency. Done means a working minimal package with the requested core, namespace, io, dataset, and parameter support.