drivendataorg / drivendataorg/deon

E.5: systematic effects (consider adding)

Open
#147 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
309
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Following up from comment thread in [PR #140](https://github.com/drivendataorg/deon/pull/140)

A few thoughts below, if we think this is worth adding. E1 is related but this feels like a sufficiently different question to consider, especially with everything that's already present in E1.

Possible wording:

E.5 Systematic effects
(could also be, say E.3 and push the other two to 4/5)

Have we considered risks posed by the scale, speed, or rigidity of the deployed model that aren't present in the equivalent human or prototype process (e.g., reinforced outcomes and feedback loops, ability to consider missing variables, societal impacts)?

Possible examples:

I think this example of feedback loop (formerly of "concept drift") actually fits in here; this is a result of using the model, not just a distribution that happens to shift on its own

- text: Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.
url: https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/
- text: -- Related academic study.
url: https://arxiv.org/abs/1706.09847

curious if folks have thoughts on other examples to represent broader systemic impacts that might not be captured elsewhere in the checklist.
possible example: speed of misinformation spread + twitter ([Science article](https://www.science.org/doi/full/10.1126/science.aap9559), Twitter's [crisis misinformation policy to slow down viral tweets](https://www.axios.com/2022/05/19/twitter-crisis-misinformation-policy), etc.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.