Layout prompt returns low precision coordinates causing ±9px errors
- Dominant language
- Python
- Stars
- 9k
- Forks
- 775
- PR merge metrics
- No merged PRs in 30d
Description
## Low Precision Dataset Issue with Layout Model
The `layout` part of the model was trained on a very low precision dataset, resulting in inaccurate bounding box coordinates.
### Problem Description
When using the prompt `Parse the reading order of this document. ...`, the typical output format is:
```md
[0.16,0.35,0.35,0.37] sec[RELATION_SEP]
[0.37,0.35,0.52,0.37] sec[PAIR_SEP]
[0.56,0.06,0.78,0.09] para[PAIR_SEP]
...
```
### Issue
The coordinates are returned with only **two decimal places of precision**. This causes significant layout errors:
- For a 896px wide document: 896px × 0.01 = **±9 pixel error margin**
- This is particularly noticeable with `y_max` values where the last character falls outside the recognized bounding box:

### Attempted Solutions
**Few-shot prompting failed:** Adding instructions in the prompt to return 3 or 4 decimal places:
```
Parse the reading order of this document.
Provide bounding box coordinates with four digits of precision.
For example, a valid output format is: [0.1232,0.5675,0.9016,0.3459] label.
Now, parse the document.
```
Does not improve the output precision.
The model continues to return only 2 decimal places regardless of prompt modifications.
### Proposed Solution
Since prompt engineering cannot resolve this precision limitation, the issue requires a **model-level fix**:
- **Fine-tune** the existing model on a higher precision dataset
- **Retrain** the model from scratch with coordinates having 3-4 decimal places
This would reduce the error margin to ±0.9px (3 decimals) or ±0.09px (4 decimals), significantly improving layout accuracy.
>Besides the precision increase to 4 digits, consider providing the **Medium** `-M` version of the model trained on twice the size: 896px * 2 = 1792px version of the documents. 896px is not enough for typical A4 documents when small fonts are used.
### Impact
The current low precision affects document parsing accuracy, especially for text elements near bounding box edges, making precise layout reconstruction difficult.
Contributor guide
No contributing guide indexed for this repository
Research direction
No repository file, test, or entry point is identified in the issue. Start by locating the layout model's training and data pipeline, then determine whether higher-precision coordinates or a Medium -M model can be supported; done means validating improved bounding-box precision on representative documents.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100