open-compass / open-compass/VLMEvalKit
`dump()` silently drops predictions that begin with a URL and exceed 2079 characters
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 2d 27m
- Merged PRs (30d)
- 18
Description
Summary
dump_xlsx in vlmeval/smp/file.py disables xlsxwriter's string→formula
conversion but not its string→URL conversion. strings_to_urls defaults to
True, so any cell whose text starts with something xlsxwriter recognises as a
URL is written as a hyperlink — and if that text is longer than Excel's
2079-character URL limit, xlsxwriter refuses to write the cell at all.
The result is an empty cell in the prediction column. No exception, no failed
run — just a UserWarning on stderr and a row whose model output is gone.
This is easy to miss because it only affects long outputs that happen to begin
with a URL, which is common in document-transcription benchmarks (OmniDocBench,
olmOCRBench) where a page's first line is often a link or a bibliography entry.
Reproduction
Independent of VLMEvalKit — pandas + xlsxwriter only, using the exact options
dump_xlsx currently passes:
import pandas as pd, tempfile, os
p = os.path.join(tempfile.mkdtemp(), "r.xlsx")
text = "http://example.com/a " + "x" * 2100 # URL first, >2079 chars
df = pd.DataFrame({"prediction": [text]})
with pd.ExcelWriter(p, engine="xlsxwriter",
engine_kwargs={"options": {"strings_to_formulas": False}}) as xw:
df.to_excel(xw, index=False)
s = pd.read_excel(p)["prediction"]
print(len(s), "rows") # 0 rows -- the only row is gone
Output:
UserWarning: Ignoring URL 'http://example.com/a xxxxx...' with link or
location/anchor > 2079 characters since it exceeds Excel's limit for URLs.
0 rows
Through dump() itself:
from vlmeval.smp import dump
import pandas as pd
text = "http://example.com/a " + "x" * 2100
dump(pd.DataFrame({"prediction": [text], "index": [0]}), "out.xlsx")
print(pd.read_excel("out.xlsx")["prediction"].isna().sum()) # 1
Threshold, measured:
| cell length | result |
|---|---|
| 2019 | present |
| 2079 | present |
| 2098 | missing |
| 2099 | missing |
Root cause
vlmeval/smp/file.py:
def dump_xlsx(data, f, **kwargs):
with pd.ExcelWriter(
f,
engine='xlsxwriter',
engine_kwargs={'options': {'strings_to_formulas': False}},
) as writer:
data.to_excel(writer, index=False)
strings_to_formulas is already disabled — so the equivalent problem with a
leading = has been encountered before — but xlsxwriter has a second
conversion of the same kind, and its default is True:
strings_to_formulas— defaultTrue, already disabled herestrings_to_urls— defaultTrue, not disabledstrings_to_numbers— defaultFalse, harmless
See xlsxwriter's
Constructor options
and write() —
write() returns -3 for a URL that exceeds the limit and writes nothing.
Impact
- Affected rows read back as
NaN, so downstream code sees a float where it
expects a string. Invlmeval/dataset/olmOCRBench/evaluator.pythis surfaces
asTypeError: write() argument must be str, not floatat
f.write(entry["md_content"]), which aborts scoring for the entire dataset —
in our case 2 bad rows out of 1403 cost the whole benchmark's score. - Evaluators that tolerate
NaNinstead score those rows as empty, silently
lowering the result with nothing to indicate why. dump()wraps its handlers inexcept Exceptionand falls back to writing a
.pkl, so a related failure can also change the output file's format with
only a warning.
Suggested fix
def dump_xlsx(data, f, **kwargs):
with pd.ExcelWriter(
f,
engine='xlsxwriter',
- engine_kwargs={'options': {'strings_to_formulas': False}},
+ engine_kwargs={'options': {'strings_to_formulas': False,
+ 'strings_to_urls': False}},
) as writer:
data.to_excel(writer, index=False)
Predictions are opaque model text, so none of xlsxwriter's "this looks like
something else" conversions are wanted for them.
Optionally, a post-write read-back comparison would catch any future coercion
of this kind rather than requiring each one to be discovered separately. If
added inside dump_xlsx, note that it should warn rather than raise — the
except Exception fallback in dump() would otherwise turn the failure into a
silent .pkl write.
Environment
- VLMEvalKit:
main - pandas 3.0.5, xlsxwriter 3.2.9 (also reproduces on older pandas 2.x)
- Behaviour comes from xlsxwriter, so it is platform-independent
Happy to open a PR if the fix looks right.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in vlmeval/smp/file.py at dump_xlsx and reproduce the issue with the provided pandas and xlsxwriter example. Update the Excel writer options so URL-like prediction text is not converted into hyperlinks, then verify that long URL-prefixed predictions remain present when the generated file is read with pandas.】【。
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100