internetarchive / internetarchive/heritrix3

WARNING: politessDelay unset, returning default 5000

Open
#553 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.3k
Forks
793
Avg merge
1d 8h
Merged PRs (30d)
8

Description

Hi!

I'm crawling with a little bit less politeness configuration than the default and I'm frequently getting (1971 times in the 12 hours I've been crawling):

```
Mar 27, 2023 10:43:53 AM org.archive.modules.CrawlURI getPolitenessDelay
WARNING: politessDelay unset, returning default 5000 for https://www.unidavi.edu.br/fiqueAtento/2023/3/pedidos-vagas-1-2023-fora-do-prazo-07 (in thread 'ToeThread #163: https://www.unidavi.edu.br/fiqueAtento/2023/3/pedidos-vagas-1-2023-fora-do-prazo-07')
```

Is this expected? The configuration rules I've modified and that are related to politeness are:

```xml




















```

Thank you!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the getPolitenessDelay entry point named in the warning and compare it with the politeness-related properties shown in the issue. Determine whether the warning is expected with this configuration and document what setting or behavior accounts for it; no source file or test is identified in the report.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.