apache / apache/datafusion-python

Remove the need for registering an ObjectStore for remote files

Abierto
#899 16 comentarios 2 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
604
Forks
174
Merge medio
1 d 7 h
PR fusionados (30 d)
4

Descripción

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently, we need to register an ObjectStore to register remote files stored in S3, https, etc. This could be more ergonomic from a DX perspective.

**Describe the solution you'd like**
Automatically detect if an ObjectStore is needed, like in the datafusion-cli; see [here](https://github.com/apache/datafusion/blob/6f8c74ca873fe015b2e7ff4eeb2f76bf21ae9d0e/datafusion-cli/src/object_storage.rs#L395).

**Describe alternatives you've considered**
keep it as it is now

**Additional context**
We should increase the test coverage for ObjectStore

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Start by comparing the Python ObjectStore registration flow with the referenced datafusion-cli implementation in datafusion-cli/src/object_storage.rs around line 395. Trace how remote S3 and HTTPS files are registered, then expand ObjectStore test coverage and verify that registration is detected automatically where needed.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
backend, data-engineering
Tipo de issue
Nueva funcionalidad
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.