huggingface / huggingface/sentence-transformers
Longest sequence and truncation of sentence
- Dominant language
- Python
- Stars
- 19.1k
- Forks
- 2.9k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 61
Description
Hi,
I wonder how the maximum length is set before getting an embedding given a sentence. Let s be a sentence such as s = [x1, x2, x3, ----, xN]. Is there a maximum length parameter n such that if N>n, then all tokens in indices above n are removed? s would be mapped to map(s) = [x1, x2, ---,xn] (This what we can see often in BERT-like models).
From this code:
```python
longest_seq = 0
for idx in length_sorted_idx[batch_start: batch_end]:
sentence = sentences[idx]
tokens = self.tokenize(sentence)
longest_seq = max(longest_seq, len(tokens))
batch_tokens.append(tokens)
features = {}
for text in batch_tokens:
sentence_features = self.get_sentence_features(text, longest_seq)
```
I am confused about what **get_sentence_features** does which is defined here (I do not get what **_first_module** corresponds to actually):
```python
def get_sentence_features(self, *features):
return self._first_module().get_sentence_features(*features)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Python code shown in the issue, especially get_sentence_features and _first_module, then trace how tokenize, longest_seq, and get_sentence_features are implemented. Document how maximum sequence length and truncation are handled, and explain what _first_module refers to so the behavior is clear from the project documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100