bytedance / bytedance/Dolphin

Layout prompt returns low precision coordinates causing ±9px errors

Open
#83 4 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9k
Forks
775
PR merge metrics
No merged PRs in 30d

Description

## Low Precision Dataset Issue with Layout Model

The `layout` part of the model was trained on a very low precision dataset, resulting in inaccurate bounding box coordinates.

### Problem Description

When using the prompt `Parse the reading order of this document. ...`, the typical output format is:

```md
[0.16,0.35,0.35,0.37] sec[RELATION_SEP]
[0.37,0.35,0.52,0.37] sec[PAIR_SEP]
[0.56,0.06,0.78,0.09] para[PAIR_SEP]
...
```
### Issue

The coordinates are returned with only **two decimal places of precision**. This causes significant layout errors:

- For a 896px wide document: 896px × 0.01 = **±9 pixel error margin**
- This is particularly noticeable with `y_max` values where the last character falls outside the recognized bounding box:

![Image](https://github.com/user-attachments/assets/1027b415-dd75-47ac-b07a-2acdf9360fe5)

### Attempted Solutions

**Few-shot prompting failed:** Adding instructions in the prompt to return 3 or 4 decimal places:
```
Parse the reading order of this document.
Provide bounding box coordinates with four digits of precision.
For example, a valid output format is: [0.1232,0.5675,0.9016,0.3459] label.
Now, parse the document.
```
Does not improve the output precision.
The model continues to return only 2 decimal places regardless of prompt modifications.

### Proposed Solution

Since prompt engineering cannot resolve this precision limitation, the issue requires a **model-level fix**:

- **Fine-tune** the existing model on a higher precision dataset
- **Retrain** the model from scratch with coordinates having 3-4 decimal places

This would reduce the error margin to ±0.9px (3 decimals) or ±0.09px (4 decimals), significantly improving layout accuracy.

>Besides the precision increase to 4 digits, consider providing the **Medium** `-M` version of the model trained on twice the size: 896px * 2 = 1792px version of the documents. 896px is not enough for typical A4 documents when small fonts are used.

### Impact

The current low precision affects document parsing accuracy, especially for text elements near bounding box edges, making precise layout reconstruction difficult.

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository file, test, or entry point is identified in the issue. Start by locating the layout model's training and data pipeline, then determine whether higher-precision coordinates or a Medium -M model can be supported; done means validating improved bounding-box precision on representative documents.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.