InseeFrLab / InseeFrLab/Trevas-Jupyter
Allow specifying options in load/write functions
- Dominant language
- Java
- Stars
- 2
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Currently, one can load/write data only with specific default options without being able to customize them.
For instancce, in [loadCsv](https://github.com/InseeFrLab/Trevas-Jupyter/blob/main/src/main/java/fr/insee/trevas/jupyter/SparkUtils.java#L70) or writeCsv I cannot specify a different separator from ";". There are [many other options relevant for reading/writing CSV files](https://spark.apache.org/docs/latest/sql-data-sources-csv.html).
loadParquet/writeParquet does not allow to specify options, such as compression (cf. [here](https://spark.apache.org/docs/latest/sql-data-sources-parquet.html)).
I propose to have additional functions, e.g. loadCsvWithParameters that takes as input the path and a Map which allows to specify any options. I am not exactly sure how one can pass a Map in VTL initialized with data. Alternatively one can provide simply a "config" Dataset.class which contains in a dataset with two columns (key,value).
Contributor guide
Assessment
This issue has not been assessed yet.