googleapis / googleapis/python-aiplatform

tests.system.vertexai.test_tokenization.TestTokenization: many tests failed

Open
#6,627 11 comments 0 reactions 0 assignees View on GitHub
api: vertex-ai flakybot: issue priority: p1 type: bug
Dominant language
Python
Stars
905
Forks
465
Avg merge
1d 13h
Merged PRs (30d)
44

Description

Many tests failed at the same time in this package.

* I will close this issue when there are no more failures in this package _and_
there is at least one pass.
* No new issues will be filed for this package until this issue is closed.
* If there are already issues for individual test cases, I will close them when
the corresponding test passes. You can close them earlier, if you prefer, and
I won't reopen them while this issue is still open.

Here are the tests that failed:
* test_count_tokens_local[_get_tokenizer_for_model_preview-gemini-1.5-pro-002-udhr-udhr-PROD_ENDPOINT] (#4624)
* test_count_tokens_local[get_tokenizer_for_model-gemini-1.5-pro-002-udhr-udhr-PROD_ENDPOINT]
* test_compute_tokens[_get_tokenizer_for_model_preview-gemini-1.5-pro-002-udhr-udhr-PROD_ENDPOINT]
* test_compute_tokens[get_tokenizer_for_model-gemini-1.5-pro-002-udhr-udhr-PROD_ENDPOINT] (#4625)
* test_count_tokens_system_instruction[gemini-1.5-pro-002-PROD_ENDPOINT]
* test_count_tokens_system_instruction_is_function_call[gemini-1.5-pro-002-PROD_ENDPOINT]
* test_count_tokens_system_instruction_is_function_response[gemini-1.5-pro-002-PROD_ENDPOINT]
* test_count_tokens_tool_is_function_declaration[gemini-1.5-pro-002-PROD_ENDPOINT]
* test_count_tokens_content_is_function_call[gemini-1.5-pro-002-PROD_ENDPOINT]
* test_count_tokens_content_is_function_response[gemini-1.5-pro-002-PROD_ENDPOINT]

-----
commit: ac5a5e4dd07890d07a553218d42040a6136fbb41
buildURL: [Build Status](https://source.cloud.google.com/results/invocations/0826b007-d9af-4e60-8584-d03c9cac927a), [Sponge](http://sponge2/0826b007-d9af-4e60-8584-d03c9cac927a)
status: failed

Contributor guide

Open the contributing guide

Research direction

Start with tests.system.vertexai.test_tokenization.TestTokenization and reproduce the listed failures at commit ac5a5e4dd07890d07a553218d42040a6136fbb41. Review the failed tokenization cases for gemini-1.5-pro-002, including system instructions, function calls, function responses, and tools. Done means this package has no failures and at least one passing test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.