[FEATURE] Support `tensorflow` to access Lance table view Lance REST service
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 286
Description
### Describe the feature
The following example is what Lance supports to run a tensorflow to access Lance tables
```python
import tensorflow as tf
import lance
# Create tf dataset
ds = lance.tf.data.from_lance("s3://my-bucket/my-dataset")
# Chain tf dataset with other tf primitives
for batch in ds.shuffling(32).map(lambda x: tf.io.decode_png(x["image"])):
print(batch)
```
It's not very elegant and hard to maintain.
```
import tensorflow as tf
import lance
import lance.tf.data
import lance_namespace as ln
ns = ln.connect("rest", {"uri": "http://localhost:9101/lance"})
ds = lance.tf.data.from_lance(
namespace=ns,
table_id=["lance_catalog", "schema", "my_table52"],
batch_size=128,
columns=["id", "value"],
filter="id > 10",
ignore_namespace_table_storage_options=False,
)
ds....
```
### Motivation
_No response_
### Describe the solution
_No response_
### Additional context
_No response_
Contributor guide
Research direction
The issue names no repository files, tests, or entry points, so first clarify the intended TensorFlow integration and how Lance REST access should be exposed. Done should support the shown namespace, table_id, batch_size, columns, filter, and storage-options arguments while preserving TensorFlow dataset chaining.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- api, data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100