cloudfour / cloudfour/lighthouse-parade

Why does sitemap.xml get crawled before the passed URL

Open
#40 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
TypeScript
Stars
373
Forks
15
Avg merge
13h 36m
Merged PRs (30d)
34

Description

Ex:

```
$ lighthouse-parade htts://www.baptistjax.com
Created CSV file
Starting the crawl...
Crawled https://www.baptistjax.com/sitemap.xml [text/xml] (646288 bytes)
Crawled https://www.baptistjax.com/ [text/html; charset=utf-8] (289135 bytes)
Report is done for https://www.baptistjax.com/
Wrote report for https://www.baptistjax.com/
Crawled https://www.baptistjax.com/services [text/html; charset=utf-8] (246368 bytes)
Report is done for https://www.baptistjax.com/services
Wrote report for https://www.baptistjax.com/services
Crawled https://www.baptistjax.com/site-search [text/html; charset=utf-8] (244996 bytes)
```

Notice that `sitemap.xml' is crawled before the url I requested. Why? Maybe this is some internal logic of simplecrawler?

Possibly related to #3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.