python / python/cpython

`csv.Sniffer.sniff()` raises `Could not determine delimiter` when the sample contains a field larger than field_size_limit()

Open
#157,678 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stdlib type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:
Summary

csv.Sniffer().sniff() raises csv.Error: Could not determine delimiter for a sample whose first row contains a field longer than the current csv.field_size_limit() (131072 by default), even when every following row is consistently delimited.

Reproduction Code
import csv 
 
sample = 'a,"' + 'x' * 200_000 + '",b\n' + 'c,d,e\n' * 50 
print(csv.Sniffer().sniff(sample).delimiter) 
Actual Behavior
Traceback (most recent call last): 
  File "repro.py", line 4, in <module> 
    print(csv.Sniffer().sniff(sample).delimiter) 
          ~~~~~~~~~~~~~~~~~~~^^^^^^^^ 
  File "Lib/csv.py", line 384, in sniff 
    raise Error("Could not determine delimiter") 
_csv.Error: Could not determine delimiter 
Expected Behavior

sniff() should return a dialect with `delimiter=','

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs
  • gh-157689

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the sample from the issue, then inspect Lib/csv.py around Sniffer.sniff(). Check how the field_size_limit affects delimiter detection and verify that the sample returns a dialect whose delimiter is a comma without weakening the limit behavior. The issue's linked PR gh-157689 indicates that work may already be underway.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.