AbsaOSS / AbsaOSS/spline

DataSource information enchancement

Offen
#1,095 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
feature
Vorherrschende Sprache
Scala
Sterne
667
Forks
155
Ø Merge
3 T. 6 Std.
Gemergte PRs (30 T.)
4

Beschreibung

### Discussed in https://github.com/AbsaOSS/spline/discussions/1093

Originally posted by **vishalag001** July 18, 2022
Currently, the **dataSource** collection only contains URI and the name is dependent on the URI(anything after the '/'). However a dataSource should ideally have a tableName, schema and related details.

In Spline, such information is captured on the write operation. Say for a hiveTable write, we have params which contains the tableName, Schema name, etc. For BigQuery, we get datasetName, projectName and tableName.

Is it possible to leverage the **operation** collection to enhance the dataSource collection ?

**Benefits of this approach:**

- The UI could refer to the schema.tableName rather than the name(which is derived from URI) and make it more meaningful.
- It will help to list dataSource URI which fall under same tableName( ,i.e, same table but different partitions)
- On UI, the list of different tables can be displayed and from there on one can navigate to the lineageOverview (by the corresponding progress Event). In case of more than 10 partitions, we can use latest partitions to display the lineage

@wajda let me know your thoughts. I am happy to contribute to this.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.