karpathy / karpathy/makemore

[Suggestion] Add a note about the training of Bengio et al. MLP

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4.3k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

Hi @karpathy, thanks for that great repo!

Maybe it would be better to note in your code that while you're training by [minimizing the CE loss](https://github.com/karpathy/makemore/blob/f61811b994280cb12ddae15ef5800baa2e3a1ca4/makemore.py#L392), Bengio actually **maximized** the log-likelihood. I know that it is equivalent in this case (one-hot vectors as ground-truth), but that's not the case in general, so maybe better to note. Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.