apache / apache/datafusion-python

Attach runtime before calling scan for foreign table provider

Aperta
#1,146 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Python
Stelle
604
Fork
174
Merge medio
1g 7h
PR unite (30g)
4

Descrizione

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

In the table provider for `lance` in the `scan` method I sometimes have to do a small amount of I/O to load dataset metadata if the metadata is not already in the cache. The `scan` method is async so I assume this is ok. However, I use `spawn_blocking` to trigger this I/O and it fails because the runtime is not active.

**Describe the solution you'd like**

The runtime should be attached before calling `scan`.

**Describe alternatives you've considered**

Currently I work around this by forcing the metadata to be cached before creating the table provider but this is inconvenient.

**Additional context**

Can provide additional details (stack traces, lance reproducers, etc.) on request.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start at the table provider's async `scan` entry point and the `spawn_blocking` path described in the issue. Reproduce a Lance metadata cache miss, trace when the runtime is attached relative to `scan`, and confirm the cache-miss path completes without the runtime error.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
databases
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.