DEFAULT_MAX_NEW_TOKENS = 1024 != MAX_INPUT_TOKEN_LENGTH = int(os.getenv("MAX_INPUT_TOKEN_LENGTH", "4096"))

Looking at app.py under demo folder I see the following code:
MAX_MAX_NEW_TOKENS = 2048
DEFAULT_MAX_NEW_TOKENS = 1024

MAX_INPUT_TOKEN_LENGTH = int(os.getenv("MAX_INPUT_TOKEN_LENGTH", "4096"))

This could lead to an overflow issue.

Implications
This discrepancy might lead to:
Truncated output: If the model generates tokens beyond the max_new_tokens limit, the output might be truncated, potentially losing important information.
Inconsistent behavior: Depending on the model's implementation, exceeding max_new_tokens might cause errors or unexpected behavior.

I suggest a mitigation strategy:
To address this issue:
Increase DEFAULT_MAX_NEW_TOKENS: Consider raising the default value to match or exceed MAX_INPUT_TOKEN_LENGTH.
Dynamic max_new_tokens calculation: Calculate max_new_tokens based on input length to ensure sufficient tokens for generation.
Input length validation: Validate input length before processing and raise an error if it exceeds MAX_INPUT_TOKEN_LENGTH.

Read demo/app.py, starting with the MAX_NEW_TOKENS and MAX_INPUT_TOKEN_LENGTH definitions and the generation path that uses them. Trace how input length and max_new_tokens are handled; done means the selected limits are consistent and oversized inputs or generated outputs have defined behavior.

Written by the indexing model from the issue text.