scrapinghub / scrapinghub/dateparser

importing dateparser is too slow

Open
#1,051 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.9k
Forks
520
Avg merge
22h 56m
Merged PRs (30d)
6

Description

hi!

first, thanks for this awesome project, it's really useful and powerful and i am grateful to not have to write this stuff myself. :)

i open this issue because I feel there's some inherent performance issue to be paid whenever we even load the dateparser library:

anarcat@curie:undertime(main)$ multitime -n 10 -s 0 -q python3 -c "import dateparser"
===> multitime results
1: -q python3 -c "import dateparser"
            Mean        Std.Dev.    Min         Median      Max
real        0.328       0.008       0.319       0.326       0.350       
user        0.313       0.009       0.299       0.315       0.331       
sys         0.013       0.008       0.000       0.014       0.028       

compare with similar libraries:

anarcat@curie:undertime(main)$ multitime -n 10 -s 0 -q python3 -c "import parsedatetime"
===> multitime results
1: -q python3 -c "import parsedatetime"
            Mean        Std.Dev.    Min         Median      Max
real        0.072       0.008       0.069       0.070       0.096       
user        0.065       0.011       0.050       0.062       0.095       
sys         0.008       0.005       0.000       0.008       0.019       
anarcat@curie:undertime(main)$ multitime -n 10 -s 0 -q python3 -c "import arrow"===> multitime results
1: -q python3 -c "import arrow"
            Mean        Std.Dev.    Min         Median      Max
real        0.064       0.006       0.061       0.062       0.081       
user        0.055       0.006       0.042       0.054       0.064       
sys         0.009       0.006       0.000       0.010       0.019       

a quick profiling seems to show it spends an inordinate amount of time compiling regular expressiongs:

anarcat@curie:undertime(main)$ python3 -m cProfile -s cumulative <(echo "import dateparser") | head -50
         627961 function calls (598845 primitive calls) in 0.558 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        9    0.000    0.000    0.593    0.066 __init__.py:1(<module>)
     60/1    0.000    0.000    0.558    0.558 {built-in method builtins.exec}
        1    0.000    0.000    0.558    0.558 63:1(<module>)
     73/1    0.000    0.000    0.558    0.558 <frozen importlib._bootstrap>:1002(_find_and_load)
     73/1    0.000    0.000    0.558    0.558 <frozen importlib._bootstrap>:967(_find_and_load_unlocked)
     70/1    0.000    0.000    0.557    0.557 <frozen importlib._bootstrap>:659(_load_unlocked)
     58/1    0.000    0.000    0.557    0.557 <frozen importlib._bootstrap_external>:784(exec_module)
     93/1    0.000    0.000    0.557    0.557 <frozen importlib._bootstrap>:220(_call_with_frames_removed)
        1    0.000    0.000    0.556    0.556 date.py:1(<module>)
        1    0.000    0.000    0.519    0.519 date_parser.py:1(<module>)
        1    0.000    0.000    0.500    0.500 timezone_parser.py:1(<module>)
     1901    0.034    0.000    0.485    0.000 regex.py:451(_compile)
      795    0.004    0.000    0.477    0.001 regex.py:349(compile)
      770    0.002    0.000    0.357    0.000 timezone_parser.py:56(build_tz_offsets)
      769    0.006    0.000    0.345    0.000 timezone_parser.py:58(get_offset)
 2255/755    0.008    0.000    0.159    0.000 _regex_core.py:382(_parse_pattern)

basically, it seems we're spending a lot of time compiling regular expressions. individually, those don't matter so much (percall=1ms) but we seem to be doing hundreds of those. I think it might be related to the timezone_parser.py file (build_tz_offsets?) but i stopped digging there.

the exact source is a little besides the point: shouldn't just importing the module be safe enough, performance wise? i know we load a default parser, but that's not what's eating us here, but rather a bunch of globals in timezone_parser.py... it seems to me those could be lazily loaded, at least?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by profiling import dateparser, then inspect timezone_parser.py, especially build_tz_offsets and get_offset, which the report identifies as likely sources of repeated regular-expression compilation. The work is done when importing dateparser avoids unnecessary startup compilation and the existing test suite still passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.