aws / aws/sagemaker-python-sdk

Add configuration of loggers for PySpark

Ouverte
#4,082 1 commentaire 1 réaction 1 personne assignée Réclamée par @nargokul Voir sur GitHub
component: processing type: logging/error reporting
Langage dominant
Python
Étoiles
2.3k
Forks
1.3k
Merge moyen
1 j 22 h
PR mergées (30 j)
35

Description

**Describe the feature you'd like**
Currently, everything for PySpark (Processor, training etc.) is logged at INFO level, including basic setup of cluster, for example:
```
yarn.resourcemanager.hostname 10.0.215.164 The hostname of the RM. yarn.nodemanager.hostname algo-1 The hostname of the NM. yarn.nodemanager.webapp.address algo-1:8042 yarn.nodemanager.vmem-pmem-ratio 5 Ratio between virtual memory to physical memory. yarn.resourcemanager.am.max-attempts 1 The maximum number of application attempts. yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,YARN_HOME,AWS_CONTAINER_CREDENTIALS_RELATIVE_URI,AWS_REGION Environment variable whitelist yarn.scheduler.minimum-allocation-mb 1 yarn.scheduler.maximum-allocation-mb 32768 yarn.scheduler.minimum-allocation-vcores 1 yarn.scheduler.maximum-allocation-vcores 8 yarn.nodemanager.resource.memory-mb 32768 yarn.nodemanager.resource.cpu-vcores 8
--
 



```
This results in a lot of CloudWatch logs, with major downsides:
- most of the logs are completely useless, printing internal details of infrastructure
- searching logs for anything actually important is very hard, which is very problematic for training and monitoring mission-critical models
- it is quite costly for running short jobs, making CloudWatch costs high compared to actual compute costs

Adding options to configure log4j logger, or at least some options to limit this (e.g. minimal logging level), would be to get rid of this. It is also very simple to implement.

**How would this feature be used? Please describe.**
Additional argument(s) passed to e.g. `PySparkProcessor`.

**Describe alternatives you've considered**
Full customizability is not necessarily required, but setting minimal log level is very important.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.