cdgriffith / cdgriffith/puremagic
Kind of long detection for some HTML files | CSV Sniffer sniffs too much data
- Dominant language
- Python
- Stars
- 242
- Forks
- 45
- PR merge metrics
- No merged PRs in 30d
Description
I worked on some old files and saw that some Excel files took weirdly long to detect the mime type using puremagic.
Upon further inspection it was actually an html export from an excel file saved with the .xls extension.
When I further analyzed why the puremagic call on this 2MiB file took over 500 seconds, I observed following "call stack":
For main.py->magic_file()
```
magic_file: file_details took 0.0030357837677001953
magic_file: identify_all took 0.0017850399017333984
magic_file: run_deep_scan took 519.073059797287
```
And inside main.py->run_deep_scan() and their sub functions
```
single_deep_scan: 0.0010013580322265625
catch_all_deep_scan: 519.0719368457794
```
With the catch all deep scan using the text_scanner which used the csv_scanner which was the culprit:
```
text_scanner: read head (1000000 bytes) took 0.0020
text_scanner: nul check took 0.0001
text_scanner: decode_any (cp1252) took 0.0087
csv_check: splitlines (9 lines) took 0.0021
csv_check: strip blank lines (5 lines left) took 0.0000
csv_check: delimiter scoring [] took 0.0183
csv_check: csv.Sniffer (None) took 519.0303
text_scanner: csv_check took 519.0510
```
It seems like inside csv_check you sent up to one MB to the csv.Sniffer() which is the culprit of the long runtime and seems like very much data for a csv sniffer. Many references like the [official documentation](https://docs.python.org/3/library/csv.html#csv.Sniffer) use 1kB of data, and theoretically a couple of lines should be sufficient.
As your csv_check already has split the up to 1MB head of the file into lines, why not set up a smaller sample to be sniffed like for example:
```
sample = "\n".join(lines[:50])[:8192]
try:
dialect = csv.Sniffer().sniff(sample, delimiters="".join(potential_delimiters))
```
which would use up to 50 lines and 8k characters increasing the execution speed to just a couple of milliseconds in my tests.
Attached you find a dummy file that takes 30 seconds with your approach vs 0.008 seconds with the sample approach.
Keep in mind this file is an html file so it should raise an `_csv.Error: Could not determine delimiter`
[intergalactic_incident_log.xls](https://github.com/user-attachments/files/31181017/intergalactic_incident_log.xls)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in main.py at run_deep_scan() and follow catch_all_deep_scan() into text_scanner and csv_check, where csv.Sniffer receives the large sample. Reproduce the delay with the attached HTML-exported .xls file, then verify that sniffing a smaller sample avoids the slowdown while still raising _csv.Error for the HTML content. Run the existing test suite and add or update coverage for this case if the relevant tests are present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 76/100