CGI-FR / CGI-FR/LINO

feat: separate structure and profile in tables.yaml

Open
#337 0 comments 0 reactions 1 assignee Claimed by @adrienaury View on GitHub
enhancement
Dominant language
Go
Stars
23
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Problem

`tables.yaml` file mixes 2 types of informations
1. information about the datasource structure (tables names, primary keys, dbinfos types)
2. information about extract or load operations (list of columns, export format, import format)

This example :

```yaml
version: "1"
tables:
- name: "film"
keys: ["film_id"]
columns:
- name: "film_id"
dbinfo:
type: "bigserial"
- name: "title"
dbinfo:
type: "varchar"
length: 30
bytes: true
- name: "picture"
export: "presence"
import: "file"
dbinfo:
type: "BLOB"
```

Contains information about the datasource structure (tables names, primary keys, dbinfos types) :

```yaml
version: "1"
tables:
- name: "film"
keys: ["film_id"]
columns:
- name: "film_id"
dbinfo:
type: "bigserial"
- name: "title"
dbinfo:
type: "varchar"
length: 30
bytes: true
- name: "picture"
dbinfo:
type: "BLOB"
```

And information about extract or load operations (list of columns to export, export formats, import formats) :

```yaml
version: "1"
tables:
- name: "film"
columns:
- name: "film_id"
- name: "title"
- name: "picture"
export: "presence"
import: "file"
```

There is a difference between each type of information
1. information about the datasource structure never change
2. information about extract or load operations will vary depending on the use case

Therefore, it would be interresting to separate these concerns in different files.

## Solution

This does not impact existing configurations.

Information about extract or load operations should be managed by the existing **ingress-descriptor** configuration. This configuration is loaded by the `pull` and `push` command via the existing flag : `--ingress-descriptor` or `-i `.

Ingress descriptor file already manage list of columns to select. The only missing information to complete extract/load operations is the import/export formats.

When using the `--ingress-descriptor` flag, import/export formats contained inside the ingress-descriptor file will be **overriding** informations loaded from the root table.yaml file. This is for retro-compatibility with current behavior.

The previous exemple could be configured like this :

**`tables.yaml`**

```yaml
version: "1"
tables:
- name: "film"
keys: ["film_id"]
columns:
- name: "film_id"
dbinfo:
type: "bigserial"
- name: "title"
dbinfo:
type: "varchar"
length: 30
bytes: true
- name: "picture"
dbinfo:
type: "BLOB"
```

**`ingress-descriptor.yaml`**

```yaml
version: v1
IngressDescriptor:
startTable: "film"
select: ["film_id", "title", "picture"]
formats:
- columns: "picture"
export: "presence"
import: "file"
```

The following command would extract data with list of columns to export and export formats defined in `ingress-descriptor.yaml`

```console
$ lino pull source --ingress-descriptor ingress-descriptor.yaml
```

The following command would load data with list of columns to import and importformats defined in `ingress-descriptor.yaml`

```console
$ lino push source --ingress-descriptor ingress-descriptor.yaml
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.