Azure / Azure/azure-kusto-python

Suggestion: Could ingest_* methods return something

Aperta
#191 3 commenti 2 reazioni 0 assegnatari Vedi su GitHub
Discussion enhancement
Lingua principale
Python
Stelle
204
Fork
118
Merge medio
13g 7h
PR unite (30g)
1

Descrizione

**Is your feature request related to a problem? Please describe.**
ingestion_ methods make developer life difficult to write robust code. They currently silently fail even if their ingestion ends up in the ".show ingestion failures". Users have no way of monitoring this as all underlying ids are hidden from the user to even check against that table.

**Describe the solution you'd like**
have ingestion_ methods return something useful to handle failures programmatically. Ideally mimicking the robust C# api.

**Describe alternatives you've considered**
I guess KIT?

edit1: So it appears our friend the KIT library has a pattern with ingestion tag monitoring: https://github.com/Azure/azure-kusto-ingestion-tools/blob/a2a256a09a66aacfe9c4756b2b0b457014013c4a/kit/kit/backends/kusto.py#L183

However the problem with ingestion tag monitoring is that it only can identify successful rows, at least with how it's implemented in KIT currently. However, the API that KIT offers seems much more full featured. Why aren't some of KIT's API features available for ingestion? seems like most of KIT's methods lend themselves well to the ingestion library. KIT makes sense standalone for CLI and data schema inferencing, but it has some nice repeated patterns anyone using the ingestion library would eventually need to write

edit2: ehhh kit is pretty tangled up with this manifest business. Possibly have ingest return the url? https://github.com/Azure/azure-kusto-python/blob/d81a549a86a5463db1c131357c179a8f3830d075/azure-kusto-ingest/azure/kusto/ingest/_ingest_client.py#L64 This would at least allow users to inspect the failures table for their specific error filtering by the random GUID used in the dataframe/file helper methods? That would effectively be the unique key

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia leggendo azure/kusto/ingest/_ingest_client.py e la gestione degli ingestion-tag in kit/kit/backends/kusto.py. Confronta i metodi di ingestion esistenti con i pattern KIT indicati e verifica come vengono identificati i fallimenti. Il lavoro è completato quando il progetto dispone di un contratto di ritorno definito e documentato che consenta ai chiamanti di monitorare o correlare programmaticamente i fallimenti di ingestion.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
azure, python
Ambito
api, data-engineering
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.