huggingface / huggingface/sentence-transformers

Longest sequence and truncation of sentence

Open
#181 7 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.1k
Forks
2.9k
Avg merge
1d 19h
Merged PRs (30d)
61

Description

Hi,
I wonder how the maximum length is set before getting an embedding given a sentence. Let s be a sentence such as s = [x1, x2, x3, ----, xN]. Is there a maximum length parameter n such that if N>n, then all tokens in indices above n are removed? s would be mapped to map(s) = [x1, x2, ---,xn] (This what we can see often in BERT-like models).

From this code:
```python
longest_seq = 0

for idx in length_sorted_idx[batch_start: batch_end]:
sentence = sentences[idx]
tokens = self.tokenize(sentence)
longest_seq = max(longest_seq, len(tokens))
batch_tokens.append(tokens)

features = {}
for text in batch_tokens:
sentence_features = self.get_sentence_features(text, longest_seq)

```
I am confused about what **get_sentence_features** does which is defined here (I do not get what **_first_module** corresponds to actually):

```python
def get_sentence_features(self, *features):
return self._first_module().get_sentence_features(*features)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Python code shown in the issue, especially get_sentence_features and _first_module, then trace how tokenize, longest_seq, and get_sentence_features are implemented. Document how maximum sequence length and truncation are handled, and explain what _first_module refers to so the behavior is clear from the project documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.