@paullizer is already working on this.

Since Feb 6, 2026.

## Chunking Strategy

Simple Chat processes a wide range of document types using a consistent chunking strategy to enable optimal context handling and LLM performance. Below is an overview of how we chunk content depending on the file type:

### General Principles

- **Chunk Size**: Targeting \~400 words of meaningful content per chunk. Some formats (HTML, Markdown) require larger text windows (up to 1200 words) due to formatting overhead that inflates token count.
- **Minimum Chunk Size**: Chunks are merged if they contain fewer than 600 words, ensuring minimal context fragmentation.
- **Table Handling**: When chunking might split tables, we replicate the table header in each chunk to preserve readability.
- **Code Blocks** (Markdown): Ensure any split code blocks retain full formatting (\`\`\`\`\`) in each chunk to maintain integrity.

---

### File-Type Specific Chunking

#### **PDF**

- Sent to Document Intelligence for OCR + layout parsing.
- If PDF is more than 500 MB or 2000 pages, it is broken into 500‐page parts.
- Each part is sent separately to Document Intelligence.
- All chunks from each part are saved under the original document in AI Search; parts exist only to work around service limits.
- **Chunked by page**.

#### **DOCX**

- Sent to Document Intelligence.
- **Chunked by \~400 words**, approximating an A4 page.

#### **DOC / DOCM**

- Processed with Python package docx2txt
- **Chunked by \~400 words**, approximating an A4 page.

#### **PPTX**

- Sent to Document Intelligence.
- **Chunked by slide (page)**.

#### **Images** (`.jpg`, `.jpeg`, `.png`, `.bmp`, `.tiff`, `.tif`, `.heif`)

- Sent to Document Intelligence for OCR.
- **One chunk per image**.

#### **TXT**

- Processed using regex word splitting.
- **Chunked by 400 words**.
- See `process_txt_file`.

#### **HTML**

- Uses `RecursiveCharacterTextSplitter`.
- **Header-based chunking**:
- Initially split by `

` tags.
- Chunks >1200 words are recursively split using `

` → `

` → ... `

`.
- **Tables**: If a table spans chunks, ensure headers are repeated per chunk.
- **Minimum chunk size**: Merge chunks <600 words into preceding ones.
- **Goal**: Maintain 400 words of *informational* content per chunk, accounting for token inflation from HTML tags.
- See `process_html_file`.

#### **Markdown (.md)**

- Uses `MarkdownHeaderTextSplitter`.
- Initial split by `#` headers (h1 → h5).
- Chunks >1200 words undergo recursive splitting.
- **Table & Code Block Handling**:
- Tables: Re-add headers if split.
- Code: Wrap with code block syntax (\`\`\`\`\`) if a split occurs.
- Minimum 600-word chunks enforced.
- See `process_md_file`.

#### **JSON**

- Uses `RecursiveJsonSplitter` - a specialized splitter designed for JSON data structures.
- **Structural splitting**:
- Understands JSON objects, arrays, and nesting
- `max_chunk_size=4000` characters
- `convert_lists=True` - intelligently handles JSON arrays
- **Maintains validity**: Each chunk is valid, parseable JSON.
- **Empty chunk filtering**: Skips trivial chunks like `{}`, `[]`, or empty strings.
- **Goal**: Preserve JSON structure while creating semantically meaningful chunks that respect object/array boundaries.
- See `process_json`.

#### **XML**

- Uses `RecursiveCharacterTextSplitter` with XML-aware separators.
- **Structure-preserving chunking**:
- Separators prioritized: `\n\n` → `\n` → `>` (end of XML tags) → space → character
- Splits at logical boundaries to maintain tag integrity
- **Chunked by 4000 characters**.
- **Goal**: Preserve XML structure by splitting at tag boundaries rather than mid-element, ensuring chunks are more semantically meaningful for LLM processing.
- See `process_xml`.

#### **YAML / YML**

- Uses `RecursiveCharacterTextSplitter` with YAML-aware separators.
- **Structure-preserving chunking**:
- Separators prioritized: `\n\n` → `\n` → `- ` (YAML list items) → space → character
- Splits at logical boundaries to maintain YAML structure
- **Chunked by 4000 characters**.
- **Goal**: Preserve YAML hierarchy and list structures by splitting at section boundaries and list items rather than mid-key or mid-value.
- See `process_yaml`.

#### **LOG**

- Processed using line-based chunking to maintain log record integrity.
- **Never splits mid-line** to preserve complete log entries.
- **Line-Level Chunking**:
1. Split file by lines using `splitlines(keepends=True)` to preserve line endings.
2. Accumulate complete lines until reaching target word count ≈1000 words.
3. When adding next line would exceed target AND chunk already has content:
- Finalize current chunk
- Start new chunk with current line
4. If single line exceeds target, it gets its own chunk to prevent infinite loops.
5. Emit chunks with complete log records.
- **Goal**: Provide substantial log context (1000 words) while ensuring no log entry is split across chunks.
- See `process_log`.

#### **Tabular (CSV, XLSX, XLS, XLSM)**

- **CSV**:
- Parsed with pandas.
- **Chunked by rows** into \~800-character chunks (1+ full rows per chunk).
- Header row prepended to each chunk.
- **XLSX / XLS**:
- Each worksheet is treated as a separate file:
- Filename format: `filename-tabname.ext`
- **Chunked by rows** (\~800 characters).
- Header rows are preserved.
- Excel formulas are evaluated to return only the computed **value**.
- See `process_tabular_file`.

#### **Video Files**

- **Transcript Extraction**:
- Use Azure Video Indexer to generate a full transcript with timestamps and, if needed, confidence scores.
- Retrieve the index JSON via the **Get Video Index** API; inspect the `insights.transcript` array for line segments with `instances[start/end]`.

* **Line-Level Chunking** (preferred for simplicity):
1. Iterate `insights.transcript`, splitting each segment’s `text` into words.

2. Accumulate segments until the aggregate is approx 30 seconds long.

3. Emit a chunk with:

- `startTime` = first segment’s `instances[0].start`.
- `text` = concatenation of segment texts.

4. Reset word count and continue.

5. See `process_video_file`.

**Audio Files**

- **Transcript Extraction (Azure Speech Services)**:

- Use Azure Speech Services Speech-to-Text REST API to generate a full transcript for audio files, including timestamps and confidence scores for each word or phrase.
- Submit your audio file to the Speech Services endpoint and retrieve the resulting transcription JSON, which provides either:
- A line/phrase-level transcript ("recognizedPhrases" with "offset"/"duration"), or

- **Line-Level Chunking**:
1. Parse the transcription JSON to obtain an array of phrase segments, each containing `text`, `offset` (start time), and `duration`.
2. Split each segment’s `text` into words.
3. Accumulate segments until the total word count ≈400.
4. Emit a chunk with:
- `startTime` = the first segment’s `offset` formatted as `HH:MM:SS.sss`.
- `text` = concatenation of segment texts.
5. Reset count and continue until all segments are processed.
6. See `process_audio_file`.