apache / apache/datafusion-python
Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface
- Linguagem predominante
- Python
- Estrelas
- 604
- Forks
- 174
- Merge médio
- 1d 7h
- PRs com merge (30d)
- 4
Descrição
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
PyArrow is a massive dependency. Unpacked, it tends to be >100MB in size, and, until the latest versions (I think?) also required numpy as its own non-optional dependency.
It's also, in effect the only current dependency
https://github.com/apache/datafusion-python/blob/f0bbad7543717c5f08ba2acb92d42c9d30fd2355/pyproject.toml#L46
It would be great if we could remove it, and that would greatly lessen the minimal environment size for datafusion python.
[Many other Python Arrow libraries](https://github.com/apache/arrow/issues/39195#issuecomment-2245718008) implement the PyCapsule Interface, so the user can use nanoarrow, arro3, Polars, DuckDB, etc, or pyarrow. Whatever is best for them.
**Describe the solution you'd like**
The Arrow PyCapsule Interface is a lightweight, decentralized protocol for sharing Arrow data between Python libraries. We already implement the PyCapsule Interface, so it's just a matter of removing places where we hard-code use of pyarrow.
**Describe alternatives you've considered**
Keep pyarrow dependency.
**Additional context**
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
Comece inspecionando pyproject.toml na linha 46 e localizando os usos restantes de pyarrow codificados diretamente nos bindings de Python. Verifique como o suporte existente a Arrow PyCapsule Interface é usado nesses caminhos. O trabalho estará concluído quando pyarrow não for mais necessário e os caminhos afetados continuarem a oferecer suporte a provedores de dados Arrow.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- python
- Domínio
- data-engineering
- Tipo de issue
- Funcionalidade
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Estagnada
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 35/100