aws / aws/amazon-sagemaker-examples
[Content Improvement] XGBOOST churn example wrong data
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
[**Link to the notebook**](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn_outputs.ipynb)
**What aspects of the notebook can be improved?**
The results described don't match with the plot in the following sections:



**What are your suggestions?**
Change the descriptions as follows:
1. Of the 247 churners, we've correctly predicted 238 of them (true positives). We also incorrectly predicted 18 customers would churn who then ended up not doing so (false positives). There are also 9 customers who ended up churning, that we predicted would not (false negatives).
2. We can see that lowering the cutoff from 0.5 to 0.3 results in 6 more true positive, 3 more false positives, and 6 fewer false negative. The numbers are small overall here, but that's 1.8% of customers overall that are shifting because of a change to the cutoff. Was this the right decision? We may end up retaining 6 extra customers, but we also unnecessarily incentivized 20 more customers who would have stayed anyway. Determining optimal cutoffs is a key step in properly applying machine learning in a real-world setting. Let's discuss this more broadly and then apply a specific, hypothetical solution for our current problem.
3. The above chart shows how picking a threshold too low results in costs skyrocketing as all customers are given a retention incentive. Meanwhile, setting the threshold too high results in too many lost customers, which ultimately grows to be nearly as costly. The overall cost can be minimized at $30k circa by setting the cutoff to 0.46, which is substantially better than the **$ Unkown value?** we would expect to lose by not taking any action.
Contributor guide
Assessment
This issue has not been assessed yet.