italia / italia/publiccode-parser-go

Set a descriptive User-Agent for outgoing HTTP requests

Open
#421 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
23
Forks
10
Avg merge
2h 55m
Merged PRs (30d)
18

Description

The parser sends Go's stdlib default `User-Agent` when checking URL reachability, making its traffic indistinguishable from any other Go program. Sites behind a WAF challenge it, producing false "not reachable" errors on valid files.

`internal/safehttp.go` builds the transport without setting headers, and `parser.go` calls `http.NewRequestWithContext` without a `User-Agent`. Captured on the wire from v5.4.3:

```
GET /
user-agent: Go-http-client/2.0
accept-encoding: gzip
```

(`Go-http-client/1.1` over HTTP/1.1; `2.0` over HTTP/2 since `ForceAttemptHTTP2` is set. No `Accept` header either.)

### Impact

Adding a `publiccode.yml` to [xwiki-platform](https://github.com/xwiki/xwiki-platform/pull/5352), the parser reports four errors on URLs that are correct and publicly reachable in a browser:

```
error: landingURL: not reachable: HTTP GET failed for https://www.xwiki.org/: forbidden resource
error: roadmap: ... forbidden resource
error: description.en.documentation: ... forbidden resource
error: description.en.apiDocumentation: ... forbidden resource
```

The host is behind Cloudflare, which serves a managed challenge (`cf-mitigated: challenge`, 403). Probing it by user agent:

| User-Agent | Status |
|---|---|
| `curl/8.7.1` | 200 |
| `Go-http-client/2.0` | 403 |
| `Wget/1.21` | 403 |
| `python-requests/2.31.0` | 403 |

The only fix available to the site administrator is to allowlist `Go-http-client/*` — the default UA of every Go program, and trivially spoofable. That is an unreasonably broad exception to ask a security team for. Any adopter behind a WAF or bot-protection layer should hit this.

### Proposal

Send a descriptive default, e.g. `publiccode-parser-go/5.4.3 (+https://github.com/italia/publiccode-parser-go)`, so administrators can write a narrow rule naming this tool. Ideally overridable via `ParserConfig`, so harvesters embedding the library can identify themselves.

Happy to open a PR if you agree on the approach and the exact string.

---
Investigated and written with [Claude Code](https://claude.com/claude-code).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with internal/safehttp.go and parser.go, where the transport and outgoing requests are built. Review how ParserConfig could provide an override and decide the descriptive default User-Agent format. Done means requests identify the parser clearly, embedding harvesters can use the intended configuration, and URL reachability checks no longer send Go's generic default.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.