Auto-sentence breaker

Is this even possible? Can we with minimal a priori knowledge can we separate sentences in all languages in all scripts enough so that when combined with a Gale-Church sentence Aligner, we can get decent training and translation data within our 200 token maximum? It doesn't have to be perfect - and splitting up sentences more than they should be may be ok. The main issues are - what will we find in different languages in terms of sentence ending punctuation?

First proposal idea:

• Have knowledge of the sentence terminating characters per script
• Analyze the incoming text to determine the script (or use the script code in the language code)
• Follow some basic rule-based knowledge:
  • Refer to https://github.com/uhermjakob/utoken/tree/main for many rules that may span scripts (emails, URL's)
  • If under 200? tokens, keep parenthesis and brackets unified
• Are there abbreviations in other scripts that also use the sentence terminating character? (D. L. Moody, etc.). If we know a bit about the types of things we could find, we may be able to develop some statistics-rule based determination for sentence breaks
• Use ebible data to test out algorithms

Start by reviewing the proposed rules, the utoken rules referenced in the issue, and the Gale-Church sentence aligner context. Use ebible data to evaluate sentence breaking across scripts, including sentence-ending punctuation, abbreviations, brackets, and the 200-token limit. Done means a tested approach that produces usable sentence-alignment data, though the issue does not name files or tests.