AbsaOSS / AbsaOSS/cobrix

Need hlep in configuration

Abierto
#323 1 comentario 0 reacciones 0 asignados Ver en GitHub
question
Lenguaje dominante
Scala
Estrellas
170
Forks
96
Merge medio
57 min
PR fusionados (30 d)
2

Descripción

## Background [Optional]
Hi, Iam new to Spark and Cobrix. we have 10 EBCDIC files to process and load to Database. each file contains 1MM records and we have an SLA of 45mins to process 10 files
## Question
Hi, we have implemented Cobrix to read EBCDIC file contains 1MM records and it is taking 15 mins nearly to read and insert into database table. could you please to improve this performance and complete 10 files with this size of data in SLA of 45mins? your response and help is appreciated.
below is the load command we are using to read the file.

Dataset ds = sparkSession.read().format().option("copybook" "copybook_path").option("schma_retention_policy", "collapse_root").option("number_of_partitions", 10).load("file_name");

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.