GoogleCloudPlatform / GoogleCloudPlatform/knowledge-catalog

Spec-conformant HTML pages with properly-encoded multi-parameter URLs hand the agent a broken URL

Open Beginner friendly
#170 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
9.2k
Forks
782
Avg merge
6h 36m
Merged PRs (30d)
85

Description

### Bug
`_extract_links` in `web/fetcher.py` reads `href` attribute values directly from raw HTML without decoding HTML entities. Because valid HTML requires `&` inside attribute values to be written as `&`, a page containing something as:
```html
Search
```

causes the fetcher to surface `https://example.com/search?q=foo&lang=en` as a link. When the web agent later calls fetch_url with that string, the request is sent with a literal `&` in the query string, which many servers either reject or silently misparse, so the agent ends up fetching the wrong page or getting a 400.

Repro
Any real-world page whose query-string links are correctly HTML-encoded (which is the spec-required form) triggers this. For example, Google Search results, GitHub search pages, and most CMS-generated pages encode & as & in href attributes.

Contributor guide

Open the contributing guide

Research direction

Start in web/fetcher.py at _extract_links and reproduce the issue with the HTML example containing an encoded ampersand in the href. Verify that extracted links decode the HTML entity before being passed to fetch_url; done means multi-parameter URLs are surfaced with the intended query separators.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.