GoogleCloudPlatform / GoogleCloudPlatform/knowledge-catalog
Spec-conformant HTML pages with properly-encoded multi-parameter URLs hand the agent a broken URL
- Dominant language
- TypeScript
- Stars
- 9.2k
- Forks
- 782
- Avg merge
- 6h 36m
- Merged PRs (30d)
- 85
Description
### Bug
`_extract_links` in `web/fetcher.py` reads `href` attribute values directly from raw HTML without decoding HTML entities. Because valid HTML requires `&` inside attribute values to be written as `&`, a page containing something as:
```html
Search
```
causes the fetcher to surface `https://example.com/search?q=foo&lang=en` as a link. When the web agent later calls fetch_url with that string, the request is sent with a literal `&` in the query string, which many servers either reject or silently misparse, so the agent ends up fetching the wrong page or getting a 400.
Repro
Any real-world page whose query-string links are correctly HTML-encoded (which is the spec-required form) triggers this. For example, Google Search results, GitHub search pages, and most CMS-generated pages encode & as & in href attributes.
Contributor guide
Research direction
Start in web/fetcher.py at _extract_links and reproduce the issue with the HTML example containing an encoded ampersand in the href. Verify that extracted links decode the HTML entity before being passed to fetch_url; done means multi-parameter URLs are surfaced with the intended query separators.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100