51zero / 51zero/eel-sdk

limit parallelism of ParquetSource to reduce memory footprint

Ouverte
#382 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Scala
Étoiles
147
Forks
32
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

Hi!

I set up a ParquetSource with a JDBCSink and ran into memory issues.
The parquet files are stored in an s3 bucket and have been written by spark (snappy-compressed ~500 MByte).

Spark writes one file per partition (default=200). This causes eel to use lots of memory since many subscriptions are submitted to the executor, although I set up the stream like this `source.toDataStream().to(sink, parallelism = 1)`

In the code I found that you initialize the executor like this `val executor = Executors.newCachedThreadPool()` which creates an unbounded ThreadPool.

I did some experiments and repartitioned the spark dataframe to 1 and stored it again. Here's the comparison (see screenshots below):
memory usage of the 200 files parquet source: >1.2 Gbyte*
memory usage of the 1 file parquet source: 83 MByte constantly.

(*) I let it run on my local machine with normal DSL internet connection. On the server it ran oom pretty quickly - meaning it used more than 2GByte (my XmX setting for the app).

![image](https://user-images.githubusercontent.com/39078/40673447-1f702fae-6372-11e8-99a2-828d96b66a1f.png)

![image](https://user-images.githubusercontent.com/39078/40673462-2a1367dc-6372-11e8-8269-6ff53f90fa3e.png)

Can you think of a way to limit the amount of parallelism? Happy to provide a merge request if you point me in the right direction.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.