Support content-aware bidirectional text in LemonMarkdown and PostHog AI

Feature request

Is your feature request related to a problem?

PostHog displays multilingual user content and AI-generated Markdown that can mix left-to-right and right-to-left scripts in the same block. Browsers handle character ordering, but selecting the wrong paragraph base direction still produces confusing punctuation and inline-run placement.

A common example is a Persian-majority sentence beginning with an English product name:

PostHog یک پلتفرم تحلیل محصول قدرتمند است.

First-strong-character direction treats this as LTR because it begins with PostHog, even though the readable paragraph direction is RTL. The inverse occurs in English-majority content containing Persian, Arabic, Hebrew, or Urdu words.

This applies directly to the shared LemonMarkdown renderer and AI messages in MarkdownMessage. The latter already splits streamed content into independently memoized blocks, which is a good boundary for safe direction classification.

Describe the solution you'd like

I would be happy to prepare a small PR after maintainer feedback. A safe pilot could:

• classify each Markdown block by strong-character content majority instead of only its first strong character;
• apply semantic dir=ltr or dir=rtl and unicode-bidi: plaintext at paragraph, heading, list-item, and table-cell boundaries;
• preserve PostHog's existing alignment—direction and text alignment should remain independent;
• exclude code, URLs, paths, identifiers, and technical tokens from direction scoring;
• work with MarkdownMessage's existing per-block memoization and streaming model;
• leave pure-LTR content unchanged, with no global DOM mutation and no rewriting of stored text;
• cover Persian, Arabic, Hebrew, Urdu, mixed punctuation, lists, links, code, and streaming updates in regression tests.

The implementation could use a small local helper or the MIT-licensed BidiLens packages, depending on PostHog's dependency policy:

• Toolkit: https://github.com/CodeinScrubs/BidiLens
• Interactive examples: https://codeinscrubs.github.io/BidiLens/
• Architecture: https://github.com/CodeinScrubs/BidiLens/blob/main/docs/ARCHITECTURE.md
• Security model: https://github.com/CodeinScrubs/BidiLens/blob/main/docs/SECURITY.md
• Known limitations: https://github.com/CodeinScrubs/BidiLens/blob/main/docs/LIMITATIONS.md

The current toolkit verification includes 396 tests, a 918-case corpus, three-browser visual checks, packed consumer tests, and native platform adapters. The corpus is not yet independently/native-speaker reviewed, so PostHog-specific fixtures and maintainer review should be required before rollout.

Describe alternatives you've considered

• dir=auto: simple, but uses the first strong character and fails the product-name example above.
• globally setting RTL/LTR: breaks mixed conversations and unrelated UI.
• rewriting strings with Unicode control characters: mutates content, creates copy/paste surprises, and can introduce security concerns.

Additional context

I am proposing an opt-in or feature-flagged integration first. If the team prefers a dependency-free helper, I can adapt the same tests and behavior without adding an npm package.

Debug info

Not tied to one PostHog deployment; this is a cross-surface rendering proposal for PostHog AI and shared Markdown.

Start with frontend/src/lib/lemon-ui/LemonMarkdown/LemonMarkdown.tsx and products/posthog_ai/frontend/messages/MarkdownMessage.tsx, focusing on the existing per-block memoization and streaming boundary. Define direction scoring and semantic direction behavior for Markdown blocks, then add regression coverage for RTL scripts, mixed punctuation, lists, links, code, and streaming updates; done means pure-LTR content remains unchanged and alignment is preserved.