hellock / hellock/icrawler

Duplicate URLs?

Open
#134 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
933
Forks
181
PR merge metrics
No merged PRs in 30d

Description

Hello! I was reading the implementation and got a bit surprised in finding that there seems to be no check for duplicate URLs before adding them to the queue. Is it not needed naturally, or am I missing something? Thank you so much!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.