huggingface / huggingface/lighteval

Add jfinqa: Japanese Financial Numerical Reasoning QA

Open
#1,168 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Summary

I'd like to add **jfinqa** — a Japanese financial numerical reasoning QA benchmark — as a new task in lighteval.

## About jfinqa

- **1,000 questions** across 3 subtasks:
- Numerical Reasoning (550): Calculate growth rates, margins, ratios from financial statements
- Consistency Checking (200): Verify internal consistency of figures
- Temporal Reasoning (250): Analyze year-over-year trends
- **68 companies** from EDINET (Japan's securities filing system)
- Covers J-GAAP, IFRS, and US-GAAP accounting standards
- **HuggingFace Dataset**: [ajtgjmdjp/jfinqa](https://huggingface.co/datasets/ajtgjmdjp/jfinqa)
- **GitHub**: [ajtgjmdjp/jfinqa](https://github.com/ajtgjmdjp/jfinqa)

## Metrics

Two metrics per subtask:
1. **Exact Match** — with Japanese financial normalisation (fullwidth→halfwidth, △→minus, comma removal, NFKC)
2. **Numerical Match** — 1% relative tolerance, handles kanji multipliers (千/百万/億/兆) and unit suffixes (円/ドル/bps)

## Prior Art

- [lm-evaluation-harness PR #3570](https://github.com/EleutherAI/lm-evaluation-harness/pull/3570) (open, mergeable)

## Baselines (zero-shot, temperature=0)

| Model | Overall | Numerical | Consistency | Temporal |
|-------|---------|-----------|-------------|----------|
| GPT-4o | **87.0%** | 80.2% | 90.5% | 99.2% |
| Gemini 2.0 Flash | 80.4% | 86.2% | 83.5% | 65.2% |
| GPT-4o-mini | 67.7% | 79.3% | 83.5% | 29.6% |
| Qwen2.5-3B | 39.6% | 46.4% | 51.0% | 15.6% |

I have a PR ready — happy to adjust the implementation based on your feedback (e.g., inspect-ai format if preferred).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.