Certain characters crash bitnet model inference?

I've been working on securing the user input, escaping invalid characters, however I've encountered a few prompts which cause the llama-cli to abruptly halt:

.\llama-cli.exe --model "..\..\..\models\Llama3-8B-1.58-100B-tokens\ggml-model-i2_s.gguf" --prompt "£" --threads 2 -c 2048 -n 20 -ngl 0 --temp 0.8
...
system_info: n_threads = 2 (n_threads_batch = 2) / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |

The command abruptly halts at system_info, offering no further logs.

This also occurs for --prompt "¬" and some other advanced unicode characters.

Where as it works for normal characters:

.\llama-cli.exe --model "..\..\..\models\Llama3-8B-1.58-100B-tokens\ggml-model-i2_s.gguf" --prompt "a" --threads 2 -c 2048 -n 20 -ngl 0 --temp 0.8
...
system_info: n_threads = 2 (n_threads_batch = 2) / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |

sampler seed: 3436479236
sampler params:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> top-k -> tail-free -> typical -> top-p -> min-p -> temp-ext -> softmax -> dist
generate: n_ctx = 2048, n_batch = 2048, n_predict = 20, n_keep = 1

a, or first- 1,000 days of life, it's not that the first 1

llama_perf_sampler_print:    sampling time =       2.02 ms /    22 runs   (    0.09 ms per token, 10918.11 tokens per second)
llama_perf_context_print:        load time =    1092.28 ms
llama_perf_context_print: prompt eval time =     133.19 ms /     2 tokens (   66.60 ms per token,    15.02 tokens per second)
llama_perf_context_print:        eval time =    1831.67 ms /    19 runs   (   96.40 ms per token,    10.37 tokens per second)
llama_perf_context_print:       total time =    1970.46 ms /    21 tokens

Is this because the unicode tokens haven't been tokenized in the model? Or is there a llamma unicode fix for this issue?

I could probably filter out the incompatible characters from the user prompt input if this cannot be worked around, is there a list of incompatible characters?

Thanks

Reproduce the provided Windows llama-cli.exe commands with £, ¬, and a, then compare where execution stops. Start from the llama-cli inference path and the linked llama.cpp unicode-fix issue; done means advanced Unicode prompts no longer halt inference or the unsupported-character behavior is clearly reported.