lance-format / lance-format/lance-ray
write_lance with create mode using namespace.create_table
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 41
- Forks
- 48
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 5
Description
Currently we first write the table and then do register_table.
There are technically 2 other ways to do this:
- create_table with empty table, and then write data to it. One issue is that the table shows up before the data is fully written
- create_table which creates a running transaction to use, which allows Ray to write data and then commit the transaction at finish time.
Currently a basic namespace probably can only do 1, since the transaction part is not fully flushed out yet.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the write_lance create-mode path and the namespace.create_table entry point. Compare the empty-table and transaction-based approaches described in the issue, then clarify which behavior the basic namespace should support and what completion means for data visibility and commit timing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100