open-compass / open-compass/VLMEvalKit

`dump()` silently drops predictions that begin with a URL and exceed 2079 characters

Open Beginner friendly
#1,689 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
768
Avg merge
2d 27m
Merged PRs (30d)
18

Description

Summary

dump_xlsx in vlmeval/smp/file.py disables xlsxwriter's string→formula
conversion but not its string→URL conversion. strings_to_urls defaults to
True, so any cell whose text starts with something xlsxwriter recognises as a
URL is written as a hyperlink — and if that text is longer than Excel's
2079-character URL limit, xlsxwriter refuses to write the cell at all.

The result is an empty cell in the prediction column. No exception, no failed
run — just a UserWarning on stderr and a row whose model output is gone.

This is easy to miss because it only affects long outputs that happen to begin
with a URL, which is common in document-transcription benchmarks (OmniDocBench,
olmOCRBench) where a page's first line is often a link or a bibliography entry.

Reproduction

Independent of VLMEvalKit — pandas + xlsxwriter only, using the exact options
dump_xlsx currently passes:

import pandas as pd, tempfile, os

p = os.path.join(tempfile.mkdtemp(), "r.xlsx")
text = "http://example.com/a " + "x" * 2100        # URL first, >2079 chars
df = pd.DataFrame({"prediction": [text]})

with pd.ExcelWriter(p, engine="xlsxwriter",
                    engine_kwargs={"options": {"strings_to_formulas": False}}) as xw:
    df.to_excel(xw, index=False)

s = pd.read_excel(p)["prediction"]
print(len(s), "rows")            # 0 rows -- the only row is gone

Output:

UserWarning: Ignoring URL 'http://example.com/a xxxxx...' with link or
location/anchor > 2079 characters since it exceeds Excel's limit for URLs.
0 rows

Through dump() itself:

from vlmeval.smp import dump
import pandas as pd

text = "http://example.com/a " + "x" * 2100
dump(pd.DataFrame({"prediction": [text], "index": [0]}), "out.xlsx")
print(pd.read_excel("out.xlsx")["prediction"].isna().sum())   # 1

Threshold, measured:

cell length result
2019 present
2079 present
2098 missing
2099 missing

Root cause

vlmeval/smp/file.py:

def dump_xlsx(data, f, **kwargs):
    with pd.ExcelWriter(
        f,
        engine='xlsxwriter',
        engine_kwargs={'options': {'strings_to_formulas': False}},
    ) as writer:
        data.to_excel(writer, index=False)

strings_to_formulas is already disabled — so the equivalent problem with a
leading = has been encountered before — but xlsxwriter has a second
conversion of the same kind, and its default is True:

  • strings_to_formulas — default True, already disabled here
  • strings_to_urls — default True, not disabled
  • strings_to_numbers — default False, harmless

See xlsxwriter's
Constructor options
and write()
write() returns -3 for a URL that exceeds the limit and writes nothing.

Impact

  • Affected rows read back as NaN, so downstream code sees a float where it
    expects a string. In vlmeval/dataset/olmOCRBench/evaluator.py this surfaces
    as TypeError: write() argument must be str, not float at
    f.write(entry["md_content"]), which aborts scoring for the entire dataset —
    in our case 2 bad rows out of 1403 cost the whole benchmark's score.
  • Evaluators that tolerate NaN instead score those rows as empty, silently
    lowering the result with nothing to indicate why.
  • dump() wraps its handlers in except Exception and falls back to writing a
    .pkl, so a related failure can also change the output file's format with
    only a warning.

Suggested fix

     def dump_xlsx(data, f, **kwargs):
         with pd.ExcelWriter(
             f,
             engine='xlsxwriter',
-            engine_kwargs={'options': {'strings_to_formulas': False}},
+            engine_kwargs={'options': {'strings_to_formulas': False,
+                                       'strings_to_urls': False}},
         ) as writer:
             data.to_excel(writer, index=False)

Predictions are opaque model text, so none of xlsxwriter's "this looks like
something else" conversions are wanted for them.

Optionally, a post-write read-back comparison would catch any future coercion
of this kind rather than requiring each one to be discovered separately. If
added inside dump_xlsx, note that it should warn rather than raise — the
except Exception fallback in dump() would otherwise turn the failure into a
silent .pkl write.

Environment

  • VLMEvalKit: main
  • pandas 3.0.5, xlsxwriter 3.2.9 (also reproduces on older pandas 2.x)
  • Behaviour comes from xlsxwriter, so it is platform-independent

Happy to open a PR if the fix looks right.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in vlmeval/smp/file.py at dump_xlsx and reproduce the issue with the provided pandas and xlsxwriter example. Update the Excel writer options so URL-like prediction text is not converted into hyperlinks, then verify that long URL-prefixed predictions remain present when the generated file is read with pandas.】【。

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.