Intermittent abort in test_trainer_with_normalizer under pytest --parallel-threads 4 (free-threaded 3.14, riscv64)

Describe the bug

Running the free-threading test command from your own wheel.yml (pytest -v --parallel-threads 4), the suite intermittently aborts in test_trainer_with_normalizer on linux riscv64 with free-threaded CPython 3.14. It is not deterministic: the same job passed on an earlier run the same morning and failed on the re-run.

I could not reproduce it on x86_64 with the same Python version, so I cannot tell you whether this is riscv64-specific or a latent race that x86's stronger memory ordering hides. I am not asking you to support riscv64. I am reporting it because the test is unmarked and this is the command your CI runs.

To Reproduce

Environment where it fails:

• linux riscv64, Ubuntu 24.04 (CI runner provided by the RISE project)
• CPython 3.14.6 free-threaded
• sentencepiece 0.2.2, built from the v0.2.2 tag with cibuildwheel 3.4.0 in quay.io/pypa/manylinux_2_39_riscv64
• pytest 9.1.1, pytest-run-parallel 0.9.1
• from python/: pytest -v --parallel-threads 4

Tail of the run:

test/sentencepiece_test.py::TestSentencepieceProcessor::test_train_kwargs PARALLEL PASSED [ 95%]
Fatal Python error: Aborted

Current thread's C stack trace (most recent call first):
  Binary file ".../.venv/bin/python3", at _Py_DumpStack+0x50 [0x2ac261f8f6]

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg (total: 2)
Aborted (core dumped) pytest -v --parallel-threads 4
test/sentencepiece_test.py::TestSentencepieceProcessor::test_trainer_with_normalizer

Full log: https://github.com/riseproject-dev/python-wheels/actions/runs/30362326241

The C stack has a single frame, so it says nothing useful. I can rebuild with symbols and get a proper backtrace under gdb if that would help.

What I checked on x86_64

CPython 3.14.0 free-threaded, the PyPI cp314t wheel of 0.2.2, pytest 9.1.1, pytest-run-parallel 0.10.0:

• test_trainer_with_normalizer alone, --parallel-threads 4: 25 runs, all passed
• the full sentencepiece_test.py, --parallel-threads 4: 7 runs, all 42 tests passed each time

Not conclusive, but it does rule out "3.14t instead of the 3.13t you test" as the whole story.

Expected behavior

The test is not marked thread_unsafe and gives each thread its own model_prefix through threading.get_native_id(), so it reads as intended to run in parallel.

Additional context

I hit this while adding riscv64 wheel builds for the RISE project. I have riscv64 boards and CI runners, so I am happy to run whatever diagnostic you want, including a debug build under gdb or ThreadSanitizer.

One thing I want to raise myself, in case it saves you time: per #1134 the policy here is NumPy-style experimental free-threading, const methods thread-safe and non-const ones not. SentencePieceTrainer.Train is not a const operation, so marking this test thread_unsafe the way test_nbest_timeout already is would be a perfectly reasonable answer, and I would take that as settled rather than push back on it.

Start with test/sentencepiece_test.py, especially TestSentencepieceProcessor::test_trainer_with_normalizer, and compare its parallel behavior with test_nbest_timeout and the free-threading command in wheel.yml. Re-run from python on the stated riscv64 environment, then use the project’s free-threading policy to determine whether the test should be marked thread_unsafe or whether a reproducible defect needs further diagnostics; done means the CI command no longer aborts intermittently.