aws / aws/sagemaker-python-sdk

Add configuration of loggers for PySpark

Open
#4,082 1 comment 1 reaction 1 assignee Claimed by @nargokul View on GitHub
component: processing type: logging/error reporting
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the feature you'd like**
Currently, everything for PySpark (Processor, training etc.) is logged at INFO level, including basic setup of cluster, for example:
```
yarn.resourcemanager.hostname 10.0.215.164 The hostname of the RM. yarn.nodemanager.hostname algo-1 The hostname of the NM. yarn.nodemanager.webapp.address algo-1:8042 yarn.nodemanager.vmem-pmem-ratio 5 Ratio between virtual memory to physical memory. yarn.resourcemanager.am.max-attempts 1 The maximum number of application attempts. yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,YARN_HOME,AWS_CONTAINER_CREDENTIALS_RELATIVE_URI,AWS_REGION Environment variable whitelist yarn.scheduler.minimum-allocation-mb 1 yarn.scheduler.maximum-allocation-mb 32768 yarn.scheduler.minimum-allocation-vcores 1 yarn.scheduler.maximum-allocation-vcores 8 yarn.nodemanager.resource.memory-mb 32768 yarn.nodemanager.resource.cpu-vcores 8
--
 



```
This results in a lot of CloudWatch logs, with major downsides:
- most of the logs are completely useless, printing internal details of infrastructure
- searching logs for anything actually important is very hard, which is very problematic for training and monitoring mission-critical models
- it is quite costly for running short jobs, making CloudWatch costs high compared to actual compute costs

Adding options to configure log4j logger, or at least some options to limit this (e.g. minimal logging level), would be to get rid of this. It is also very simple to implement.

**How would this feature be used? Please describe.**
Additional argument(s) passed to e.g. `PySparkProcessor`.

**Describe alternatives you've considered**
Full customizability is not necessarily required, but setting minimal log level is very important.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.