ageron / ageron/handson-ml2

Question on Chapter 18 - loss functions

Open
#604 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
30k
Forks
13.1k
PR merge metrics
No merged PRs in 30d

Description

Greetings, I'm working through the cartpole example on page 695 of the third edition, and I have a question about the code presented:

```
def play_one_step(env, obs, model, loss_fn):
with tf.GradientTape() as tape:
left_proba = model(obs[np.newaxis])
action = (tf.random.uniform([1, 1]) > left_proba)
y_target = tf.constant([[1.]]) - tf.cast(action, tf.float32)
loss = tf.reduce_mean(loss_fn(y_target, left_proba))

grads = tape.gradient(loss, model.trainable_variables)
obs, reward, done, truncated, info = env.step(int(action))
return obs, reward, done, truncated, grads
```
I'm confused about y_target, and why it's an input into the loss function. If the action is False (0), y_target is 1. If the action is True (1), y_target is 0. It seems like we are effectively saying that the model should have been more confident in whatever it's output was. Is that the correct way to think about what y_target is accomplishing? If so, is there something happening in a later step where we're determining if the action recommended by the model was beneficial?

I have similar questions about the loss function presented on page 710, but if I can get some clarification on this earlier example, perhaps I'll understand the more challenging Q-value example.

Thank you!

Contributor guide

No contributing guide indexed for this repository

Research direction

Read the CartPole example in Chapter 18, page 695, including play_one_step and its loss function, then compare it with the Q-value loss discussion on page 710. Done means providing a clear explanation of y_target, how the loss is used, and where the example determines whether an action was beneficial.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, tensorflow
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.