aws / aws/sagemaker-python-sdk
Add configuration of loggers for PySpark
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 35
Description
**Describe the feature you'd like**
Currently, everything for PySpark (Processor, training etc.) is logged at INFO level, including basic setup of cluster, for example:
```
yarn.resourcemanager.hostname 10.0.215.164 The hostname of the RM. yarn.nodemanager.hostname algo-1 The hostname of the NM. yarn.nodemanager.webapp.address algo-1:8042 yarn.nodemanager.vmem-pmem-ratio 5 Ratio between virtual memory to physical memory. yarn.resourcemanager.am.max-attempts 1 The maximum number of application attempts. yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,YARN_HOME,AWS_CONTAINER_CREDENTIALS_RELATIVE_URI,AWS_REGION Environment variable whitelist yarn.scheduler.minimum-allocation-mb 1 yarn.scheduler.maximum-allocation-mb 32768 yarn.scheduler.minimum-allocation-vcores 1 yarn.scheduler.maximum-allocation-vcores 8 yarn.nodemanager.resource.memory-mb 32768 yarn.nodemanager.resource.cpu-vcores 8
--
```
This results in a lot of CloudWatch logs, with major downsides:
- most of the logs are completely useless, printing internal details of infrastructure
- searching logs for anything actually important is very hard, which is very problematic for training and monitoring mission-critical models
- it is quite costly for running short jobs, making CloudWatch costs high compared to actual compute costs
Adding options to configure log4j logger, or at least some options to limit this (e.g. minimal logging level), would be to get rid of this. It is also very simple to implement.
**How would this feature be used? Please describe.**
Additional argument(s) passed to e.g. `PySparkProcessor`.
**Describe alternatives you've considered**
Full customizability is not necessarily required, but setting minimal log level is very important.
Contributor guide
Assessment
This issue has not been assessed yet.