CGI-FR / CGI-FR/LINO

feat: parquet datasource

Open
#320 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
23
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Add a new datasource type supported by lino : parquet

The initial implementation supports :
- no ingress description definition, by default lino pull will follow the parquet schema
- a single table by file, with name "data" by convention
- read only, can't update existing file (lino push will overwrite existing file)

Use the provided example file to test parquet features : [test.zip](https://github.com/user-attachments/files/17240859/test.zip)

## lino dataconnector

```
$ lino dc add myfile 'parquet://path/to/test.parquet'
successfully added dataconnector
$ cat dataconnector.yaml
version: v1
dataconnectors:
- name: myfile
url: parquet://path/to/test.parquet
readonly: false
```

## lino table

```
$ lino table extract myfile
lino finds 1 table
$ cat tables.yaml
version: v1
tables:
- name: data
columns:
- name: col_type_binary_string
- name: col_type_int_32
- name: col_type_boolean
```

## lino pull

```
$ lino pull myfile
{"col_type_binary_string": "6FBFXK4V82XQA4EZ0UCY8PZZN2B256", "col_type_int_32": 324442164, "col_type_boolean": true}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.