Azure / Azure/azure-kusto-python

Suggestion: Could ingest_* methods return something

Offen
#191 3 Kommentare 2 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Discussion enhancement
Vorherrschende Sprache
Python
Sterne
204
Forks
118
Ø Merge
13 T. 7 Std.
Gemergte PRs (30 T.)
1

Beschreibung

**Is your feature request related to a problem? Please describe.**
ingestion_ methods make developer life difficult to write robust code. They currently silently fail even if their ingestion ends up in the ".show ingestion failures". Users have no way of monitoring this as all underlying ids are hidden from the user to even check against that table.

**Describe the solution you'd like**
have ingestion_ methods return something useful to handle failures programmatically. Ideally mimicking the robust C# api.

**Describe alternatives you've considered**
I guess KIT?

edit1: So it appears our friend the KIT library has a pattern with ingestion tag monitoring: https://github.com/Azure/azure-kusto-ingestion-tools/blob/a2a256a09a66aacfe9c4756b2b0b457014013c4a/kit/kit/backends/kusto.py#L183

However the problem with ingestion tag monitoring is that it only can identify successful rows, at least with how it's implemented in KIT currently. However, the API that KIT offers seems much more full featured. Why aren't some of KIT's API features available for ingestion? seems like most of KIT's methods lend themselves well to the ingestion library. KIT makes sense standalone for CLI and data schema inferencing, but it has some nice repeated patterns anyone using the ingestion library would eventually need to write

edit2: ehhh kit is pretty tangled up with this manifest business. Possibly have ingest return the url? https://github.com/Azure/azure-kusto-python/blob/d81a549a86a5463db1c131357c179a8f3830d075/azure-kusto-ingest/azure/kusto/ingest/_ingest_client.py#L64 This would at least allow users to inspect the failures table for their specific error filtering by the random GUID used in the dataframe/file helper methods? That would effectively be the unique key

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginnen Sie mit dem Lesen von azure/kusto/ingest/_ingest_client.py und der Verarbeitung von ingestion-Tags in kit/kit/backends/kusto.py. Vergleichen Sie die vorhandenen Ingestion-Methoden mit den referenzierten KIT-Mustern und untersuchen Sie, wie Fehler identifiziert werden. Als abgeschlossen gilt die Aufgabe, wenn das Projekt einen festgelegten, dokumentierten Rückgabevertrag hat, mit dem Aufrufer Ingestion-Fehler programmgesteuert überwachen oder korrelieren können.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
azure, python
Bereich
api, data-engineering
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.