duckdb sniff_csv fails with > 10240 lines
- Langage dominant
- Python
- Étoiles
- 187
- Forks
- 112
- Merge moyen
- 13 h 29 min
- PR mergées (30 j)
- 17
Description
### What happens?
When I use sniff_csv implicitly using read_csv with the arguments ", header=True, delim=';', sample_size=10241", it triggers an "unable to detect csv format"-error.
- I can place multiple different lines on line 20241 and they all trigger it. The lines seem fine and not different than any others.
- When I reduce the sample_size to 10240, the import works again.
- When I then remove lines 10236-10240 from the original file and keep the sample_size on 10240, it also works.
This indicates to me that 10240 is an upper limit for the sampling before it goes wrong, but the default is of 20xxx.
I unfortunately can't provide you the sample.
The duckdb-version is "v1.2.0"
### To Reproduce
```
self._db.execute(f"CREATE OR REPLACE TABLE {all_table_name} AS SELECT * FROM read_csv('{self.event.tmp_file_path}', header=True, delim=';', sample_size=10241)")
```
### OS:
Linux
### DuckDB Package Version:
1.2.0
### Python Version:
3.13.7
### Full Name:
Michel
### Affiliation:
Acme
### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
### Did you include all relevant data sets for reproducing the issue?
No - I cannot share the data sets because they are confidential
### Did you include all code required to reproduce the issue?
- [x] Yes, I have
### Did you include all relevant configuration to reproduce the issue?
- [x] Yes, I have
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par l’appel execute Python fourni en utilisant read_csv, header=True, delim=';' et les valeurs de sample_size 10240 et 10241. Étudiez pourquoi sniff_csv échoue lors de l’échantillonnage de plus de 10240 lignes ; le travail est terminé lorsque la reproduction ne génère plus d’erreur "unable to detect csv format" pour des valeurs de sample_size plus élevées.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python
- Domaine
- database
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 38/100