elastic / elastic/crawler

Support `304` and non-modified content better

Open
#221 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
224
Forks
48
Avg merge
23h 16m
Merged PRs (30d)
18

Description

### Problem Description

This came up in https://github.com/elastic/crawler/pull/219

We currently don't support checking for modified content, but we could do this. If a server returns a `304` we just drop updating the content. Because `304` responses don't contain any content, we can't re-index the content from that URL as part of the response processing. This means that if a server has previously ingested a page when it returned `200`, but that page now returns `304`, the page will be deleted from the index and the content becomes invisible to Crawler.

### Proposed Solution

Crawler should treat `304` responses in the following way:

- Save URLs for `304` in a temporary list (in-memory)
- During purge crawl phase, if the URL is related to content from already indexed into ES, don't delete the doc
- Instead, update `last_crawled_at` value
- Output this information in debug logger

Contributor guide

Open the contributing guide

Research direction

Start by tracing response processing for HTTP 304 responses and the purge crawl phase, then inspect how already-indexed Elasticsearch documents are selected for deletion. Verify the temporary URL list, preservation of related documents with an updated last_crawled_at value, and debug logging against the proposed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, ruby
Domain
backend, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.