ageron / ageron/handson-ml3

[BUG] Chapter 13: Solution to Exercise 10 might have an issue (paragraph e)

Open
#82 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
14.1k
Forks
5.3k
PR merge metrics
No merged PRs in 30d

Description

I am completely new to the material, so could be wrong, but I think the solution of exercise 10 has an issue. The solution presents the following function:
```python
def compute_mean_embedding(inputs):
not_pad = tf.math.count_nonzero(inputs, axis=-1)
n_words = tf.math.count_nonzero(not_pad, axis=-1, keepdims=True)
sqrt_n_words = tf.math.sqrt(tf.cast(n_words, tf.float32))
return tf.reduce_sum(inputs, axis=1) / sqrt_n_words
```
This function is supposed to calculate the square root of the number of words by counting non-zero vectors.
However, it is plugged into the model after the Embeddings layer, like so:
```python
embedding_size = 20
tf.random.set_seed(42)

text_vectorization = tf.keras.layers.TextVectorization(
max_tokens=max_tokens, output_mode="int")
text_vectorization.adapt(sample_reviews)

model = tf.keras.Sequential([
text_vectorization,
tf.keras.layers.Embedding(input_dim=max_tokens,
output_dim=embedding_size,
mask_zero=True), # tokens => zero vectors
tf.keras.layers.Lambda(compute_mean_embedding),
tf.keras.layers.Dense(100, activation="relu"),
tf.keras.layers.Dense(1, activation="sigmoid"),
])
```
From my tests, it does not seem that `mask_zero=True` guarantees that the padding token 0 (output of the `TextVectorization` layer) is embedded to a zeros vector by the `Embedding` layer. So in fact, I'm not sure it correctly counts the number of words. If I am correct, it always uses the square root of the length of the longest sentence in the batch.

---

Here is my suggested solution. The layer should replace both the `Embedding` and `Lambda` layers in the notebook's solution. Note, however, that when I tested a model using this layer, it did not have a better accuracy than the notebook's solution for the same dataset and hyperparameters.
```python
class SentenceEmbedding(tf.keras.layers.Layer):
def __init__(self, input_dim, output_dim, **kwargs):
super().__init__(**kwargs)
self.input_dim = input_dim
self.output_dim = output_dim
self.embedding_layer = tf.keras.layers.Embedding(
input_dim=self.input_dim,
output_dim=self.output_dim,
mask_zero=True
)

def call(self, input):
n_words = tf.math.count_nonzero(input, axis=-1, keepdims=True, dtype=tf.float32)
embeddings_output = self.embedding_layer(input)
return tf.reduce_sum(embeddings_output, axis=1) / tf.sqrt(n_words)

def get_config(self):
base_config = super().get_config()
return {
**base_config,
'input_dim': self.input_dim,
'output_dim': self.output_dim
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Chapter 13 notebook's solution to Exercise 10, paragraph e, and review the Embedding and Lambda layers shown in the issue. Check how mask_zero and the padding vectors affect the word count and run the notebook's model example to compare the behavior with the proposed layer. Done means the solution counts words as intended and the notebook explanation or code is corrected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.