apache / apache/datafusion-python
Attach runtime before calling scan for foreign table provider
- Lingua principale
- Python
- Stelle
- 604
- Fork
- 174
- Merge medio
- 1g 7h
- PR unite (30g)
- 4
Descrizione
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In the table provider for `lance` in the `scan` method I sometimes have to do a small amount of I/O to load dataset metadata if the metadata is not already in the cache. The `scan` method is async so I assume this is ok. However, I use `spawn_blocking` to trigger this I/O and it fails because the runtime is not active.
**Describe the solution you'd like**
The runtime should be attached before calling `scan`.
**Describe alternatives you've considered**
Currently I work around this by forcing the metadata to be cached before creating the table provider but this is inconvenient.
**Additional context**
Can provide additional details (stack traces, lance reproducers, etc.) on request.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Start at the table provider's async `scan` entry point and the `spawn_blocking` path described in the issue. Reproduce a Lance metadata cache miss, trace when the runtime is attached relative to `scan`, and confirm the cache-miss path completes without the runtime error.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- databases
- Tipo di issue
- Funzionalità
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 45/100