Language-specific tokenisers seem hard to use properly

It looks like seshat has support activating different tokenisers depending on the language set at index creation time:

https://github.com/matrix-org/seshat/blob/71e17fa9c53de4776ee34b8b68f3b783147a4cd2/src/index.rs#L162-L176

My concern with this is that you can easily be receiving events in several languages, so there's no way to specify a single language for all your events.

Maybe instead we should apply tokenisers per event and try to detect language per event somehow? I am not quite sure what the right model is, but there current one seems hard to use with it's assumption of a single language.

Start by reading the language-selection logic in src/index.rs around lines 162-176, then trace how tokenisers are applied when events are indexed. Define how mixed-language events should be handled and what language detection or per-event configuration would mean; done requires an agreed model and corresponding behavior for events in multiple languages.

Written by the indexing model from the issue text.