algolia / algolia/docsearch

Anchors are being stripped out (using `sitemaps`, `linkExtractor` and `externalData`)

Open
#1,831 4 comments 0 reactions 0 assignees View on GitHub
crawler
Dominant language
TypeScript
Stars
4.4k
Forks
439
Avg merge
14h 32m
Merged PRs (30d)
19

Description

## Description

We are using Algolia Crawler UI for parsing our mixed static HTML & SPA website (using hash router). All URLs are provided in `sitemaps` Crawler config.

```js
new Crawler({
startUrls: [],
sitemaps: ["https://example.com/sitemap.xml"],
// ...
})
```

## Steps to reproduce

Use a sitemap with the following content:

```xml

https://example.com/page.html
monthly
0.6

https://example.com/subpage.html#/foo
monthly
0.6

https://example.com/subpage.html#/bar
monthly
0.6

```

... or using the static `linkExtractor`:

```js
new Crawler({
// ...
linkExtractor: () => {
return [
"https://example.com/page.html",
"https://example.com/subpage.html#/foo",
"https://example.com/subpage.html#/bar",
];
},
// ...
})
```

Then run the URL Tester.

Result:
```
LINKS
Found 2 links matching your configuration
- https://example.com/page.html
- https://example.com/subpage.html
```

## Expected behavior

Expected result:
```
LINKS
Found 3 links matching your configuration
- https://example.com/page.html
- https://example.com/subpage.html#/foo
- https://example.com/subpage.html#/bar
```

Note those are not section anchors. Those are actual pages, correctly parsed in URL Tester with the `renderJavaScript: true` option when passing the full URL with the anchor.

## Environment

- Algolia Crawler UI

Similar issues:
- https://github.com/algolia/docsearch/issues/1282
- https://github.com/algolia/docsearch/issues/1823
- https://github.com/algolia/docsearch/issues/1009 (old infra)
- https://github.com/algolia/docsearch/issues/53 (rly old)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the sitemap and static linkExtractor cases in the URL Tester with the three hash-based URLs. Trace how those links are collected and normalized, including the externalData path mentioned in the title. Done means the two hash URLs remain distinct links while the existing page URL continues to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.