openai / openai/tiktoken

Fix missing line numbers in stack traces

Open
#103 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
19.3k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Should be a matter of putting debug = true in Cargo.toml.

before:

num_threads: 1, num_bytes: 100005605
tiktoken        6603873.890819601 bytes / s
huggingface     1668452.742767104 bytes / s
webtext encode  tiktoken        13205 ms
webtext encode  data_gym        31538 ms
webtext encode  huggingface     52667 ms
base64 encode   tiktoken        146 ms
base64 encode   data_gym        1004 ms
base64 encode   huggingface     470 ms

after:

num_threads: 1, num_bytes: 100005605
tiktoken        6605162.737709517 bytes / s
huggingface     1676763.3459086807 bytes / s
webtext encode  tiktoken        13016 ms
webtext encode  data_gym        31160 ms
webtext encode  huggingface     52656 ms
base64 encode   tiktoken        143 ms
base64 encode   data_gym        971 ms
base64 encode   huggingface     470 ms

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Inspect Cargo.toml and confirm how its debug setting affects generated stack traces. Set debug = true, then reproduce the reported run and verify that stack traces include line numbers; the benchmark output in the issue provides the comparison context.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.