bug: o200k_harmony duplicates special token id 200018

Summary

In o200k_harmony, two special token names share the same token id 200018:

• <|endofprompt|> → 200018
• <|reserved_200018|> → 200018 (conflict)

Token ids must be unique within an encoding.

---

Reproduction

import tiktoken
from collections import defaultdict

print(tiktoken.__version__)  # expect 0.12.0

enc = tiktoken.get_encoding("o200k_harmony")
sp = enc._special_tokens
print(sp)  # shows '<|endofprompt|>': 200018 and '<|reserved_200018|>': 200018

# Optional: explicit duplicate-id check
id2names = defaultdict(list)
for name, tid in sp.items():
    id2names[tid].append(name)
dups = {tid: names for tid, names in id2names.items() if len(names) > 1}
print(dups)  

# -> {200018: ['<|endofprompt|>', '<|reserved_200018|>']}

Actual

<|reserved_200018|> duplicates <|endofprompt|> (both id=200018).

Expected

No two special token names share the same token id.

Root Cause

tiktoken_ext/openai_public.py bulk-generates reserved_* specials for [200013, 201088) without excluding ids already used by explicit specials, introducing <|reserved_200018|> as a duplicate of <|endofprompt|>.

• This is the full code segment where the bug is introduced:
  https://github.com/openai/tiktoken/blob/97e49cbadd500b5cc9dbb51a486f0b42e6701bee/tiktoken_ext/openai_public.py#L95-L145

• At line 100, <|endofprompt|> is already registered with the id 200018:
  https://github.com/openai/tiktoken/blob/97e49cbadd500b5cc9dbb51a486f0b42e6701bee/tiktoken_ext/openai_public.py#L100

• However, at line 145, the code adds all ids from 200013 to 201088 as reserved tokens, which mistakenly includes the already-used id 200018:
  https://github.com/openai/tiktoken/blob/97e49cbadd500b5cc9dbb51a486f0b42e6701bee/tiktoken_ext/openai_public.py#L145

Fix

• Remove <|reserved_200018|> which duplicates <|endofprompt|> (both id=200018).
• When generating reserved_*, skip ids already defined in special_tokens to prevent future collisions.
• Implemented in PR #458.

Tests

Add tests/test_token_ids_unique.py to enforce token-id uniqueness across all encodings:

• special token ids are unique (no two names share the same id);
• mergeable vocab ids are unique when _mergeable_ranks is exposed.

The test fails before this change and passes after.

Compatibility

No behavior change to encoding/decoding. Only removes the duplicate special token entry.

Environment

• tiktoken version: 0.12.0
• Python: 3.13.7
• OS: macOS/Linux/Windows (reproducible)

Start in tiktoken_ext/openai_public.py around the explicit special-token definitions and reserved-token generation. Run tests/test_token_ids_unique.py and verify that all special-token IDs, and exposed mergeable vocabulary IDs, are unique; the issue notes that the work is already implemented in PR #458.