Building for size

Adding tokenizers-cpp to our project made the binary size go up from a (stripped) baseline of 1.8MB to 8.4MB in a release build on Linux x86_64. This was just with the stable rust toolchain and all defaults.

I'm no rust expert but I applied a number of the options (that I could get to work) from this page https://github.com/johnthagen/min-sized-rust and was able to trivially get the binary size down to 4.2MB. Given that rust static links by default and there is a lot of data manipulation standard library code, this doesn't completely shock me (but it is still quite large compared to our baseline).

Recording here the things I quickly tried to achieve that:

1. set(TOKENIZERS_CPP_RUST_FLAGS "-Zlocation-detail=none") in CMakeLists.txt (feature request: consider making these more configurable from the including project)
2. Use the build-std approach listed above by adding this to the cargo command line in CMakeLists: -Z build-std=std,panic_abort -Z build-std-features="optimize_for_size" (and using a nightly toolchain)
3. Add to Cargo.toml:

[profile.release]
lto = true
opt-level = "z"  # Optimize for size.
codegen-units = 1
panic = "abort"

I wasn't being super principaled, but iirc 1 and 2 combined shaved off ~500KB or so. LTO and build-std gave 2-3MB and the rest filled in the extra.

It would be good to be able to customize these things easily in the including project. Might call for some additional CMake goo and such.

Start in CMakeLists.txt by tracing TOKENIZERS_CPP_RUST_FLAGS and the Cargo command that builds the Rust component; review the referenced Cargo.toml release profile settings as well. The change should make size-related Rust build options configurable from the including project and preserve the existing build flow; compare release binary sizes to verify the result.