feat: separate structure and profile in tables.yaml
- Dominant language
- Go
- Stars
- 23
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
`tables.yaml` file mixes 2 types of informations
1. information about the datasource structure (tables names, primary keys, dbinfos types)
2. information about extract or load operations (list of columns, export format, import format)
This example :
```yaml
version: "1"
tables:
- name: "film"
keys: ["film_id"]
columns:
- name: "film_id"
dbinfo:
type: "bigserial"
- name: "title"
dbinfo:
type: "varchar"
length: 30
bytes: true
- name: "picture"
export: "presence"
import: "file"
dbinfo:
type: "BLOB"
```
Contains information about the datasource structure (tables names, primary keys, dbinfos types) :
```yaml
version: "1"
tables:
- name: "film"
keys: ["film_id"]
columns:
- name: "film_id"
dbinfo:
type: "bigserial"
- name: "title"
dbinfo:
type: "varchar"
length: 30
bytes: true
- name: "picture"
dbinfo:
type: "BLOB"
```
And information about extract or load operations (list of columns to export, export formats, import formats) :
```yaml
version: "1"
tables:
- name: "film"
columns:
- name: "film_id"
- name: "title"
- name: "picture"
export: "presence"
import: "file"
```
There is a difference between each type of information
1. information about the datasource structure never change
2. information about extract or load operations will vary depending on the use case
Therefore, it would be interresting to separate these concerns in different files.
## Solution
This does not impact existing configurations.
Information about extract or load operations should be managed by the existing **ingress-descriptor** configuration. This configuration is loaded by the `pull` and `push` command via the existing flag : `--ingress-descriptor` or `-i `.
Ingress descriptor file already manage list of columns to select. The only missing information to complete extract/load operations is the import/export formats.
When using the `--ingress-descriptor` flag, import/export formats contained inside the ingress-descriptor file will be **overriding** informations loaded from the root table.yaml file. This is for retro-compatibility with current behavior.
The previous exemple could be configured like this :
**`tables.yaml`**
```yaml
version: "1"
tables:
- name: "film"
keys: ["film_id"]
columns:
- name: "film_id"
dbinfo:
type: "bigserial"
- name: "title"
dbinfo:
type: "varchar"
length: 30
bytes: true
- name: "picture"
dbinfo:
type: "BLOB"
```
**`ingress-descriptor.yaml`**
```yaml
version: v1
IngressDescriptor:
startTable: "film"
select: ["film_id", "title", "picture"]
formats:
- columns: "picture"
export: "presence"
import: "file"
```
The following command would extract data with list of columns to export and export formats defined in `ingress-descriptor.yaml`
```console
$ lino pull source --ingress-descriptor ingress-descriptor.yaml
```
The following command would load data with list of columns to import and importformats defined in `ingress-descriptor.yaml`
```console
$ lino push source --ingress-descriptor ingress-descriptor.yaml
```
Contributor guide
Assessment
This issue has not been assessed yet.