italia / italia/publiccode-parser-go
Set a descriptive User-Agent for outgoing HTTP requests
- Dominant language
- Go
- Stars
- 23
- Forks
- 10
- Avg merge
- 2h 55m
- Merged PRs (30d)
- 18
Description
The parser sends Go's stdlib default `User-Agent` when checking URL reachability, making its traffic indistinguishable from any other Go program. Sites behind a WAF challenge it, producing false "not reachable" errors on valid files.
`internal/safehttp.go` builds the transport without setting headers, and `parser.go` calls `http.NewRequestWithContext` without a `User-Agent`. Captured on the wire from v5.4.3:
```
GET /
user-agent: Go-http-client/2.0
accept-encoding: gzip
```
(`Go-http-client/1.1` over HTTP/1.1; `2.0` over HTTP/2 since `ForceAttemptHTTP2` is set. No `Accept` header either.)
### Impact
Adding a `publiccode.yml` to [xwiki-platform](https://github.com/xwiki/xwiki-platform/pull/5352), the parser reports four errors on URLs that are correct and publicly reachable in a browser:
```
error: landingURL: not reachable: HTTP GET failed for https://www.xwiki.org/: forbidden resource
error: roadmap: ... forbidden resource
error: description.en.documentation: ... forbidden resource
error: description.en.apiDocumentation: ... forbidden resource
```
The host is behind Cloudflare, which serves a managed challenge (`cf-mitigated: challenge`, 403). Probing it by user agent:
| User-Agent | Status |
|---|---|
| `curl/8.7.1` | 200 |
| `Go-http-client/2.0` | 403 |
| `Wget/1.21` | 403 |
| `python-requests/2.31.0` | 403 |
The only fix available to the site administrator is to allowlist `Go-http-client/*` — the default UA of every Go program, and trivially spoofable. That is an unreasonably broad exception to ask a security team for. Any adopter behind a WAF or bot-protection layer should hit this.
### Proposal
Send a descriptive default, e.g. `publiccode-parser-go/5.4.3 (+https://github.com/italia/publiccode-parser-go)`, so administrators can write a narrow rule naming this tool. Ideally overridable via `ParserConfig`, so harvesters embedding the library can identify themselves.
Happy to open a PR if you agree on the approach and the exact string.
---
Investigated and written with [Claude Code](https://claude.com/claude-code).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with internal/safehttp.go and parser.go, where the transport and outgoing requests are built. Review how ParserConfig could provide an override and decide the descriptive default User-Agent format. Done means requests identify the parser clearly, embedding harvesters can use the intended configuration, and URL reachability checks no longer send Go's generic default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100