eikek / eikek/docspell

Cyrillic characters mangled in PDF text extraction due to limited Ghostscript font fallback

Open
#2,921 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Elm
Stars
2.3k
Forks
184
Avg merge
8h 2m
Merged PRs (30d)
4

Description

### Description
PDFs containing Cyrillic text (without embedded fonts) show corrupted characters after processing due to Ghostscript's limited fallback font support.

### Visual Evidence
Before processing:
![image](https://github.com/user-attachments/assets/37785696-3a1b-46da-a941-bccec66f087f)

After processing:
![image](https://github.com/user-attachments/assets/b3c911ee-aeaa-4eff-aad4-f36872b9653c)

Anonymized Test PDF:
[input.pdf](https://github.com/user-attachments/files/18427117/input.pdf)

### Technical Details
Ghostscript currently uses a fallback font with limited character support:
```/usr/share/ghostscript/10.02.1/Resource/CIDFSubst/DroidSansFallback.ttf```

### Solution
Use Noto Sans as fallback font for better Unicode coverage:

1. Download and install the font:
```bash
wget https://github.com/notofonts/notofonts.github.io/raw/refs/heads/main/fonts/NotoSans/hinted/ttf/NotoSans-Regular.ttf
sudo mkdir -p /usr/share/fonts/truetype/noto
sudo mv NotoSans-Regular.ttf /usr/share/fonts/truetype/noto/
sudo fc-cache -f -v
```

2. Configure Ghostscript:
```text
# In /usr/share/ghostscript/10.02.1/Resource/Init/cidfmap
/CIDFallBack (/usr/share/fonts/truetype/noto/NotoSans-Regular.ttf) ;
```

### Environment
- Operating System: Linux
- Ghostscript version: 10.02.1

Contributor guide

Open the contributing guide

Research direction

Start by reproducing extraction with the attached input.pdf in the Linux environment using Ghostscript 10.02.1, then inspect /usr/share/ghostscript/10.02.1/Resource/Init/cidfmap and the current DroidSansFallback.ttf path. Done means Cyrillic characters from PDFs without embedded fonts are extracted correctly with the project's processing flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.