bytedance / bytedance/Dolphin

Can't reproduce Fox-Page-En Results

Open
#84 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9k
Forks
775
PR merge metrics
No merged PRs in 30d

Description

Hello there!

Firstly, congrats and thank you for open-sourcing this excellent model!

I'm trying to reproduce your results from `Table 1` from your [paper](https://arxiv.org/pdf/2505.14059) where you report `0.0114` (1.14%) `ED`; however, I'm not able to reproduce it. I'm probably doing something wrong when aggregating element contents after parsing, but since there's no evaluation script, I can't tell.

This is what I did:

Machine Info
```
Ubuntu 22.04.3 LTS
RTX 3090
Python 3.10
```

### First -> set up the env by cloning and installing dependencies

```bash
git clone https://github.com/bytedance/Dolphin.git
cd Dolphin
pip install -r requirements.txt
```

### Second -> Download HF models and `Fox-Page-En` data

```
huggingface-cli download ByteDance/Dolphin --local-dir ./hf_model
huggingface-cli download EduardoPacheco/Fox-Page-En --local-dir fox-page-en --repo-type dataset
```

The dataset is hosted at Hugging Face Hub at [EduardoPacheco/Fox-Page-En](https://huggingface.co/datasets/EduardoPacheco/Fox-Page-En). I followed instructions from [Fox Repo](https://github.com/ucaslcl/Fox). I downloaded [focus_benchmark_test.zip](https://huggingface.co/datasets/ucaslcl/Fox_benchmark_data/blob/main/focus_benchmark_test.zip) and moved the images from `en_pdf_png` and the annotations from `en_page_ocr.json` to `EduardoPacheco/Fox-Page-En` for easy access. I think this is all correct, as you mentioned in your paper (Section 4.2 -> Page-level Evaluation -> (a) Fox-Page -> line 2) that `Fox-Page-En` has 112 English samples, and this is the number of samples I got from doing this.

### Third -> Run `Dolphin` on the images

```bash
python demo_page_hf.py --model_path ./hf_model --input_path ./fox-page-en/data/ --save_dir ./fox-page-en/results
```

### Fourth -> Evaluate predictions

This is probably where the error is. I used the following script to evaluate predictions:

```python
from pathlib import Path

# pip install pandas jiwer nltk
import nltk
import jiwer
import pandas as pd

RESULTS_DIR = Path("fox-page-en/results/recognition_json")
GT_PATH = Path("fox-page-en/en_page_ocr.json")

def load_ground_truth(file_path: Path) -> pd.DataFrame:
df = pd.read_json(file_path)
df["conversations"] = df["conversations"].apply(lambda x: x[1]["value"])
df['image'] = df['image'].apply(lambda x: Path(x).stem)
df = df.rename(columns={"conversations": "text"})
return df

def load_prediction_sample(file_path: Path) -> str:
df = pd.read_json(file_path)
# Combining element-parsed contents in reading order by adding new-line between contents
# Not sure if this is the correct way
return "\n".join(df["text"].tolist())

def load_predictions(dir_path: Path) -> pd.DataFrame:
predictions = []
for file_path in dir_path.glob("*.json"):
predictions.append((file_path.stem, load_prediction_sample(file_path)))
return pd.DataFrame(predictions, columns=["image", "text"])

gt_df = load_ground_truth(GT_PATH)
pred_df = load_predictions(RESULTS_DIR)
df = (
gt_df
.merge(
pred_df,
on="image",
how="left",
suffixes=("_gt", "_pred")
)
.dropna()
.drop(columns=["len"])
)

cer = jiwer.cer(
reference=df["text_gt"].tolist(),
hypothesis=df["text_pred"].tolist()
)

print(f"CER -> {cer:.2%}")
# Output of print statement is `CER -> 5.55%`
```

The 5.55% is quite different from 1.14%, which leads me to believe I'm doing something wrong during the evaluation.

I assume I'm either:
1. Incorrectly aggregating the parsed contents
2. Incorrectly computing the metric by not following the same text normalization (?)
3. Incorrectly computing metric using `jiwer.cer` (although I tried following [Fox](https://github.com/ucaslcl/Fox/blob/main/eval_tools/eval_ocr_test.py#L35) code and led to divergent results as well)
4. Incorrectly computing metric using micro instead of macro average (I assume you computed `ED` assuming the dataset as a unique corpus instead of taking the average of `ED` for all samples)

It would be great to have a sample script to show how to evaluate a sample for reproducibility purposes.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with demo_page_hf.py and the generated recognition_json files described in the issue, then compare the reported workflow with Fox's eval_tools/eval_ocr_test.py. Determine how predictions should be aggregated and normalized, and whether ED is computed corpus-wide or per sample. Done means a repository evaluation script or documented procedure reproduces the Fox-Page-En result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.