microsoft / microsoft/onnxruntime-inference-examples

android app token to string issue

Open
#484 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C++
Stars
1.7k
Forks
414
Avg merge
1d 6h
Merged PRs (30d)
14

Description

Hello. I am trying to run phi-3.5 ONNX on Android. I'm reaching out because I'm not sure how to resolve the issue related to token to string conversion. This occurs when I instruct it to output in a different language, and I confirm through the logs that the output is not being generated.

2024-10-23 17:06:06.369 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Generated token: 31734: 안
2024-10-23 17:06:06.369 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Time taken to generate token: 1.32 seconds
2024-10-23 17:06:06.433 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Generated token: 238: 
2024-10-23 17:06:06.433 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Time taken to generate token: 0.064 seconds
2024-10-23 17:06:06.495 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Generated token: 136: 
2024-10-23 17:06:06.495 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Time taken to generate token: 0.062 seconds
2024-10-23 17:06:06.556 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Generated token: 152: 
2024-10-23 17:06:06.556 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Time taken to generate token: 0.061 seconds
2024-10-23 17:06:06.618 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Generated token: 30944: 하
2024-10-23 17:06:06.618 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Time taken to generate token: 0.062 seconds
2024-10-23 17:06:06.679 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Generated token: 31578: 세
2024-10-23 17:06:06.679 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Time taken to generate token: 0.061 seconds
2024-10-23 17:06:06.682 29366-29366 time.genai.demo         ai.onnxruntime.genai.demo            I  Waiting for a blocking GC NativeAlloc
2024-10-23 17:06:06.692 29366-29366 time.genai.demo         ai.onnxruntime.genai.demo            I  WaitForGcToComplete blocked NativeAlloc on NativeAlloc for 10.871ms
2024-10-23 17:06:06.695 29366-29378 InputTransport          ai.onnxruntime.genai.demo            D  Input channel destroyed: 'ClientS', fd=135
2024-10-23 17:06:06.759 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Generated token: 31527: 요
2024-10-23 17:06:06.759 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Time taken to generate token: 0.08 seconds
2024-10-23 17:06:06.820 29366-29512 genai.demo.MainActivity ai.onnxruntime.genai.demo            I  Generated token: 29991: !

When I execute the phi 3.5 tokenizer in Python, the output is 안녕하세요!, but the android output is 안하세요!.
I want to decode three tokens([238, 136, 152]) to obtain the correct results. I would appreciate any guidance on how to achieve this.
Thank you.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the Android genai.demo.MainActivity token-generation and logging path, then compare its decoding of tokens [238, 136, 152] with the phi-3.5 tokenizer output in Python. Done means the Android output includes 안녕하세요! rather than 안하세요!.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, python
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.