Anchors are being stripped out (using `sitemaps`, `linkExtractor` and `externalData`)
- Dominant language
- TypeScript
- Stars
- 4.4k
- Forks
- 439
- Avg merge
- 14h 32m
- Merged PRs (30d)
- 19
Description
## Description
We are using Algolia Crawler UI for parsing our mixed static HTML & SPA website (using hash router). All URLs are provided in `sitemaps` Crawler config.
```js
new Crawler({
startUrls: [],
sitemaps: ["https://example.com/sitemap.xml"],
// ...
})
```
## Steps to reproduce
Use a sitemap with the following content:
```xml
https://example.com/page.html
monthly
0.6
https://example.com/subpage.html#/foo
monthly
0.6
https://example.com/subpage.html#/bar
monthly
0.6
```
... or using the static `linkExtractor`:
```js
new Crawler({
// ...
linkExtractor: () => {
return [
"https://example.com/page.html",
"https://example.com/subpage.html#/foo",
"https://example.com/subpage.html#/bar",
];
},
// ...
})
```
Then run the URL Tester.
Result:
```
LINKS
Found 2 links matching your configuration
- https://example.com/page.html
- https://example.com/subpage.html
```
## Expected behavior
Expected result:
```
LINKS
Found 3 links matching your configuration
- https://example.com/page.html
- https://example.com/subpage.html#/foo
- https://example.com/subpage.html#/bar
```
Note those are not section anchors. Those are actual pages, correctly parsed in URL Tester with the `renderJavaScript: true` option when passing the full URL with the anchor.
## Environment
- Algolia Crawler UI
Similar issues:
- https://github.com/algolia/docsearch/issues/1282
- https://github.com/algolia/docsearch/issues/1823
- https://github.com/algolia/docsearch/issues/1009 (old infra)
- https://github.com/algolia/docsearch/issues/53 (rly old)
Contributor guide
Research direction
Start by reproducing the sitemap and static linkExtractor cases in the URL Tester with the three hash-based URLs. Trace how those links are collected and normalized, including the externalData path mentioned in the title. Done means the two hash URLs remain distinct links while the existing page URL continues to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, typescript
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100