InseeFrLab / InseeFrLab/Trevas-Jupyter

Allow specifying options in load/write functions

Open
#108 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Currently, one can load/write data only with specific default options without being able to customize them.

For instancce, in [loadCsv](https://github.com/InseeFrLab/Trevas-Jupyter/blob/main/src/main/java/fr/insee/trevas/jupyter/SparkUtils.java#L70) or writeCsv I cannot specify a different separator from ";". There are [many other options relevant for reading/writing CSV files](https://spark.apache.org/docs/latest/sql-data-sources-csv.html).

loadParquet/writeParquet does not allow to specify options, such as compression (cf. [here](https://spark.apache.org/docs/latest/sql-data-sources-parquet.html)).

I propose to have additional functions, e.g. loadCsvWithParameters that takes as input the path and a Map which allows to specify any options. I am not exactly sure how one can pass a Map in VTL initialized with data. Alternatively one can provide simply a "config" Dataset.class which contains in a dataset with two columns (key,value).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.