unclecode / unclecode/crawl4ai
[Bug]: BFSDeepCrawlStrategy.can_process_url() rejects valid single-label hostnames (netloc without a dot)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 83.9k
- Forks
- 8.7k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 11
Description
crawl4ai version
0.9.2
Expected Behavior
Single-label/internal hostnames without a dot should be treated as valid
hostnames, at least when the URL has a valid scheme (http/https) and
non-empty netloc. The dot-check appears to be an overly strict heuristic
that isn't part of the documented API/config (no allow_single_label_hosts
or similar flag exists).
Current Behavior
- Depth 0 (start URL) is fetched successfully.
- Every discovered link at depth > 0 is logged as:
Invalid URL: https://name/xyz/xyz, error: Invalid domain
The configuredDomainFilteris never even reached, since the exception
is raised beforefilter_chain.apply()is called.
Is this reproducible?
Yes
Inputs Causing the Bug
Steps to Reproduce
1. Start URL: `https://name/xyz/` (a host with no dot in its name,
reachable e.g. via internal DNS or /etc/hosts).
2. Configure `BFSDeepCrawlStrategy` with
`filter_chain=FilterChain([DomainFilter(allowed_domains=["iltis3"])])`.
3. Run `crawler.arun()`.
Code snippets
OS
Linux
Python version
3.12.13
Browser
No response
Browser version
No response
Error logs & Screenshots (if applicable)
[INIT].... → Crawl4AI 0.9.2
[FETCH]... ↓ https://name/xyz | ✓ | ⏱: 0.91s
[SCRAPE].. ◆ https://name/xyz | ✓ | ⏱: 0.06s
[EXTRACT]. ■ https://name/xyz | ✓ | ⏱: 0.03s
[COMPLETE] ● https://name/xyz | ✓ | ⏱: 1.01s
Invalid URL: https://name/xyz/neu-im-intranet, error: Invalid domain
Invalid URL: name/xyz/suche, error: Invalid domain
Invalid URL: https://name/xyz
/allgemein/stellen, error: Invalid domain
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating BFSDeepCrawlStrategy.can_process_url() and trace how discovered links reach filter_chain.apply(); reproduce the failure with https://name/xyz/ and a single-label hostname. Done means valid http/https URLs with a non-empty netloc are accepted and the configured DomainFilter is reached for links beyond depth 0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- web-dev
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100