CoreBPE::new has an unconstrained NSE type parameter, making normal downstream calls fail type inference

Verified on current main at 08a5f3b (May 24, 2026).

CoreBPE::new currently has this signature in src/lib.rs:

pub fn new<E, SE, NSE>(
    encoder: E,
    special_tokens_encoder: SE,
    pattern: &str,
) -> Result<Self, Box<dyn std::error::Error + Send + Sync>>
where
    E: IntoIterator<Item = (Vec<u8>, Rank)>,
    SE: IntoIterator<Item = (String, Rank)>,
    NSE: IntoIterator<Item = (String, (Rank, Rank))>,

The NSE type parameter is not used in the function arguments or return type, so normal downstream calls cannot infer it.

Minimal repro from an external crate:

use tiktoken::{CoreBPE, Rank};

fn main() {
    let _ = CoreBPE::new(
        vec![(b"a".to_vec(), 0 as Rank)],
        Vec::<(String, Rank)>::new(),
        r".",
    );
}

cargo check fails with:

error[E0283]: type annotations needed
 --> src/main.rs:4:13
  |
4 |     let _ = CoreBPE::new(
  |             ^^^^^^^^^^^^ cannot infer type of the type parameter `NSE` declared on the associated function `new`

This seems like a real public Rust API bug rather than an internal-only issue, since Cargo.toml publishes an rlib in addition to the Python extension.

Expected behavior:

• A normal downstream call to CoreBPE::new(...) should compile without requiring unrelated generic annotations.

Likely fix:

• Remove the unused NSE type parameter from CoreBPE::new.

Thanks!

Start in src/lib.rs at the CoreBPE::new signature and inspect the external-crate reproduction from the issue. Remove the unused generic constraint, then run cargo check with the shown CoreBPE::new call; done means normal downstream calls compile without unrelated type annotations.