Can't reproduce Fox-Page-En Results
- Dominant language
- Python
- Stars
- 9k
- Forks
- 775
- PR merge metrics
- No merged PRs in 30d
Description
Hello there!
Firstly, congrats and thank you for open-sourcing this excellent model!
I'm trying to reproduce your results from `Table 1` from your [paper](https://arxiv.org/pdf/2505.14059) where you report `0.0114` (1.14%) `ED`; however, I'm not able to reproduce it. I'm probably doing something wrong when aggregating element contents after parsing, but since there's no evaluation script, I can't tell.
This is what I did:
Machine Info
```
Ubuntu 22.04.3 LTS
RTX 3090
Python 3.10
```
### First -> set up the env by cloning and installing dependencies
```bash
git clone https://github.com/bytedance/Dolphin.git
cd Dolphin
pip install -r requirements.txt
```
### Second -> Download HF models and `Fox-Page-En` data
```
huggingface-cli download ByteDance/Dolphin --local-dir ./hf_model
huggingface-cli download EduardoPacheco/Fox-Page-En --local-dir fox-page-en --repo-type dataset
```
The dataset is hosted at Hugging Face Hub at [EduardoPacheco/Fox-Page-En](https://huggingface.co/datasets/EduardoPacheco/Fox-Page-En). I followed instructions from [Fox Repo](https://github.com/ucaslcl/Fox). I downloaded [focus_benchmark_test.zip](https://huggingface.co/datasets/ucaslcl/Fox_benchmark_data/blob/main/focus_benchmark_test.zip) and moved the images from `en_pdf_png` and the annotations from `en_page_ocr.json` to `EduardoPacheco/Fox-Page-En` for easy access. I think this is all correct, as you mentioned in your paper (Section 4.2 -> Page-level Evaluation -> (a) Fox-Page -> line 2) that `Fox-Page-En` has 112 English samples, and this is the number of samples I got from doing this.
### Third -> Run `Dolphin` on the images
```bash
python demo_page_hf.py --model_path ./hf_model --input_path ./fox-page-en/data/ --save_dir ./fox-page-en/results
```
### Fourth -> Evaluate predictions
This is probably where the error is. I used the following script to evaluate predictions:
```python
from pathlib import Path
# pip install pandas jiwer nltk
import nltk
import jiwer
import pandas as pd
RESULTS_DIR = Path("fox-page-en/results/recognition_json")
GT_PATH = Path("fox-page-en/en_page_ocr.json")
def load_ground_truth(file_path: Path) -> pd.DataFrame:
df = pd.read_json(file_path)
df["conversations"] = df["conversations"].apply(lambda x: x[1]["value"])
df['image'] = df['image'].apply(lambda x: Path(x).stem)
df = df.rename(columns={"conversations": "text"})
return df
def load_prediction_sample(file_path: Path) -> str:
df = pd.read_json(file_path)
# Combining element-parsed contents in reading order by adding new-line between contents
# Not sure if this is the correct way
return "\n".join(df["text"].tolist())
def load_predictions(dir_path: Path) -> pd.DataFrame:
predictions = []
for file_path in dir_path.glob("*.json"):
predictions.append((file_path.stem, load_prediction_sample(file_path)))
return pd.DataFrame(predictions, columns=["image", "text"])
gt_df = load_ground_truth(GT_PATH)
pred_df = load_predictions(RESULTS_DIR)
df = (
gt_df
.merge(
pred_df,
on="image",
how="left",
suffixes=("_gt", "_pred")
)
.dropna()
.drop(columns=["len"])
)
cer = jiwer.cer(
reference=df["text_gt"].tolist(),
hypothesis=df["text_pred"].tolist()
)
print(f"CER -> {cer:.2%}")
# Output of print statement is `CER -> 5.55%`
```
The 5.55% is quite different from 1.14%, which leads me to believe I'm doing something wrong during the evaluation.
I assume I'm either:
1. Incorrectly aggregating the parsed contents
2. Incorrectly computing the metric by not following the same text normalization (?)
3. Incorrectly computing metric using `jiwer.cer` (although I tried following [Fox](https://github.com/ucaslcl/Fox/blob/main/eval_tools/eval_ocr_test.py#L35) code and led to divergent results as well)
4. Incorrectly computing metric using micro instead of macro average (I assume you computed `ED` assuming the dataset as a unique corpus instead of taking the average of `ED` for all samples)
It would be great to have a sample script to show how to evaluate a sample for reproducibility purposes.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with demo_page_hf.py and the generated recognition_json files described in the issue, then compare the reported workflow with Fox's eval_tools/eval_ocr_test.py. Determine how predictions should be aggregated and normalized, and whether ED is computed corpus-wide or per sample. Done means a repository evaluation script or documented procedure reproduces the Fox-Page-En result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100