scrapinghub / scrapinghub/dateparser
[Discussion] Performance in search dates
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 520
- Avg merge
- 22h 56m
- Merged PRs (30d)
- 6
Description
I'm working on a Flask app which does some markup parsing. One of the things it does is it parses strings like Arriving tomorrow by 9pm or Delivered on Friday. All of the strings are in English and they are short. Today I bumped the version of dateparser from 0.7.6 to 1.0.0 and this is what I saw in the distribution metrics (p50, p95, p99) of the function calling search (function abridged):
STATUS_TEXT_DELIVERED = re.compile(r"delivered", re.IGNORECASE)
settings = {
"PREFER_DATES_FROM": "past"
if bool(STATUS_TEXT_DELIVERED.search(text))
else "future",
}
search_results = search_dates(text, languages=["en"], settings=settings)

One thing which strikes me most is huge latency spikes when the app is rebooted on deploy and how it calms down after some significant amount of time. This function is currently called around 20 times per minute, but we are expecting this number to grow to at least 400 rpm. On the screenshot you can see three deploys (red stripes).
Now, I have a very limited insight into what performance instrumentation you've been using, but what would be the easiest way to pinpoint what's happening with the search right after it starts from scratch? And why does it take so long to figure out the happy state?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the search_dates call and compare cold-start and warmed-up profiles for dateparser 0.7.6 and 1.0.0 using the short English inputs shown. The work is complete when the source of the startup latency and the reason for warm-up are identified with reproducible measurements, or the investigation is narrowed to a specific follow-up change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100