Support `304` and non-modified content better
- Dominant language
- Ruby
- Stars
- 224
- Forks
- 48
- Avg merge
- 23h 16m
- Merged PRs (30d)
- 18
Description
### Problem Description
This came up in https://github.com/elastic/crawler/pull/219
We currently don't support checking for modified content, but we could do this. If a server returns a `304` we just drop updating the content. Because `304` responses don't contain any content, we can't re-index the content from that URL as part of the response processing. This means that if a server has previously ingested a page when it returned `200`, but that page now returns `304`, the page will be deleted from the index and the content becomes invisible to Crawler.
### Proposed Solution
Crawler should treat `304` responses in the following way:
- Save URLs for `304` in a temporary list (in-memory)
- During purge crawl phase, if the URL is related to content from already indexed into ES, don't delete the doc
- Instead, update `last_crawled_at` value
- Output this information in debug logger
Contributor guide
Research direction
Start by tracing response processing for HTTP 304 responses and the purge crawl phase, then inspect how already-indexed Elasticsearch documents are selected for deletion. Verify the temporary URL list, preservation of related documents with an updated last_crawled_at value, and debug logging against the proposed behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, ruby
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100