mesa / mesa/mesa-llm

bug: MemoryEntry display crashes with UnicodeEncodeError on Windows cp125x consoles

Open Beginner friendly
#321 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
73
Forks
89
Avg merge
14d 58m
Merged PRs (30d)
2

Description

## Summary

`MemoryEntry.__str__()` embeds literal Unicode box-drawing characters (U+2514 `└`, U+2500 `─`, U+251C `├`) in its output. When `display=True` (the default), `Console().print(panel)` writes these to `sys.stdout`. On Windows installations where `sys.stdout.encoding` is a cp125x variant, these codepoints have no encoding mapping and raise `UnicodeEncodeError`, crashing the simulation.

This affects all memory backends (`STMemory`, `STLTMemory`, `LTMemory`) since they all call `MemoryEntry.display()`.

## Reproduction

```python
# repro.py — no API keys needed, no LLM calls, pure rendering
from rich.console import Console
from rich.panel import Panel

# Simulate a cp1252 stdout (default on Windows en-US when piped/redirected)
import io, sys
buf = io.BytesIO()
sys.stdout = io.TextIOWrapper(buf, encoding="cp1252", errors="strict")

# Replicate MemoryEntry.__str__() output
content = "[bold cyan][Observation][/bold cyan]\n [blue]\u2514\u2500[/blue] [cyan]price :[/cyan] $20"

panel = Panel(content, title="Step 1 | Agent 0", title_align="left",
border_style="bright_blue", padding=(0, 1))
Console().print(panel)
```

```
stdout encoding: cp1252
+- Step 1 | Test 0 ------...---+ <- Rich degrades ITS borders to ASCII (safe_box)
| [Observation] |
| CRASHES: 'charmap' codec can't encode characters in position 0-1: character maps to
```

Rich's `safe_box` correctly degrades its own panel borders to ASCII. But the box-drawing characters in `__str__()` are **literal content text**, not Rich decoration -- Rich cannot degrade them, so they pass through to `sys.stdout.write()` and fail.

## Expected behavior

Memory display renders without crashing on all platforms and all Windows code pages.

## Actual behavior

`UnicodeEncodeError: 'charmap' codec can't encode character '\u2514'` -- simulation crashes.

## Who is affected

`sys.stdout.encoding` on Windows depends on the system locale. The box-drawing characters used (`└─├`) crash on all cp125x variants:

| Encoding | Region | `└─├` (U+2514/2500/251C) |
|---|---|---|
| cp1252 | Western Europe (en, fr, de, es, pt, it, nl, ...) | CRASH |
| cp1250 | Central Europe (cs, pl, hu, sk, ro, hr) | CRASH |
| cp1251 | Cyrillic (ru, bg, uk) | CRASH |
| cp1253 | Greek | CRASH |
| cp1254 | Turkish | CRASH |
| cp1255 | Hebrew | CRASH |
| cp1256 | Arabic | CRASH |
| cp874 | Thai | CRASH |
| cp437 / cp850 | DOS/legacy US-EU console | safe |
| cp932 / cp936 / cp949 | Japanese / Chinese / Korean | safe |
| cp65001 | UTF-8 (opt-in) | safe |

This triggers in common scenarios: piping output, running inside VS Code / PyCharm / Jupyter, CI runners -- anywhere Python falls back to `locale.getpreferredencoding()` which returns the cp125x variant.

## Options considered

### 1. Replace with `•` (U+2022 bullet) -- rejected

`•` is safe in cp125x but **crashes on cp437, cp850, cp932, cp936, cp949** (DOS-legacy and Asian Windows). This trades one crash for another.

| Encoding | `└─├` | `•` (U+2022) |
|---|---|---|
| cp1252 (Western Windows) | CRASH | safe |
| cp437 (DOS/legacy US) | safe | CRASH |
| cp932 (Japanese) | safe | CRASH |
| cp936 (Chinese) | safe | CRASH |

### 2. Reconfigure `sys.stdout` to UTF-8 -- rejected

Libraries should not mutate global `sys.stdout` encoding. Side effects on user code, not guaranteed to work (buffer may not support `reconfigure`), and does not fix the root cause.

### 3. Wrap `console.print()` in try/except -- rejected

Hides the root cause, swallows the error silently, adds complexity. The user sees no output instead of a crash -- still broken, just quieter.

### 4. Replace with ASCII tree markers (`+--`, `|--`) -- recommended

`+`, `-`, `|` are pure ASCII (bytes 0x2B, 0x2D, 0x7C). Safe in every encoding ever created -- cp125x, cp437, cp9xx, cp65001, all of them. Preserves the tree structure that `└─`/`├─` provided. This is the same rendering that `tree --charset=ASCII` and `npm list` use.

Before:
```
[Observation]
└─ price : $20
└─ seller : Bob
[Action]
├─ (1)
│ └─ offer : $25
├─ (2)
```

After:
```
[Observation]
+-- price : $20
+-- seller : Bob
[Action]
|-- (1)
| +-- offer : $25
|-- (2)
```

7 lines changed in `memory.py`, zero logic change, zero behavior change.

## Proposed fix

File: `mesa_llm/memory/memory.py`, `MemoryEntry.__str__()` (lines 52-90)

Replace `└─` with `+--` and `├─` with `|--`. Add a test verifying `str(entry)` output is encodable across all major Windows code pages.

Contributor guide

Open the contributing guide

Research direction

Start in mesa_llm/memory/memory.py at MemoryEntry.__str__() (lines 52-90), then inspect how MemoryEntry.display() renders the result. Verify the replacement tree markers remain ASCII-encodable across the listed Windows code pages and add the requested str(entry) encoding test. Done means memory display renders without UnicodeEncodeError on those encodings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.