haskell / haskell/wreq

Avoid URL encoding.

Open
#152 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
381
Forks
72
PR merge metrics
No merged PRs in 30d

Description

I'm observing an issue with `wreq-0.5.4.0`, where I'd like to avoid URL encoding parameters for some servers that don't handle it very well. Say I'd like to set a parameter `foo=bar,baz`, and I'd like to send these parameters to the request without URL encoding.

There seems to be no way to do this with `wreq`. Using `param` seems to url encode the set parameters before they are sent.
Here's a repro:

``` haskell
-- set params.
λ> let opts = Wreq.defaults & Wreq.param "foo" .~ [ "bar,baz" ]
-- lets see what we have:
λ> opts
Options { manager = Left _, proxy = Nothing, auth = Nothing, headers = [("User-Agent","haskell wreq-0.5.4.0")], params = [("foo","bar,baz")], redirects = 10, cookies = Just (CJ {expose = []}) } -- no URL encoding yet , or at least not when `show` is called.

-- lets trigger an error so that the request gets logged.
λ> Wreq.getWith opts "https://apibeta.centralapp.com/api/v2/no/such/path" -- will respond with a 404.
*** Exception: HttpExceptionRequest Request {
host = "apibeta.centralapp.com"
port = 443
secure = True
requestHeaders = [("User-Agent","haskell wreq-0.5.4.0")]
path = "/api/v2/no/such/path"
queryString = "?foo=bar%2Cbaz" -- params have been URL encoded.
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
proxySecureMode = ProxySecureWithConnect
}
```

Now I've retried this with plain `http-client` with versions;
```
http-client-0.7.13.1
http-client-tls-0.3.6.1
```
And the behaviour is different:

``` haskell
-- create a request.
λ> req <- HTTP.parseRequest "https://apibeta.centralapp.com/api/v2/no/such/path"
-- set query params.
λ> HTTP.httpLbs mgr (req { HTTP.queryString = "?foo=bar,baz" })
-- make the request, also getting a 404:
λ> HTTP.httpLbs (req { HTTP.queryString = "?foo=bar,baz" }) mgr
Response {responseStatus = Status {statusCode = 404, statusMessage = "Not Found"}, responseVersion = HTTP/1.1, responseHeaders = [("Date","Thu, 31 Aug 2023 08:24:39 GMT"),("Content-Type","application/json;charset=utf-8"),("Transfer-Encoding","chunked"),("Connection","keep-alive"),("Set-Cookie","AWSALB=YR0tnOLtRO4hnEkt/k8O1g6hB1H567vUtd6GShkUk42UMX0ERWsScVfBPWqWQLugubSrEpMIhbwmgeo135PIOv9v6pipXF98ot34Ia0iL/sKiFyVmNTfkybUk7TM; Expires=Thu, 07 Sep 2023 08:24:39 GMT; Path=/"),("Set-Cookie","AWSALBCORS=YR0tnOLtRO4hnEkt/k8O1g6hB1H567vUtd6GShkUk42UMX0ERWsScVfBPWqWQLugubSrEpMIhbwmgeo135PIOv9v6pipXF98ot34Ia0iL/sKiFyVmNTfkybUk7TM; Expires=Thu, 07 Sep 2023 08:24:39 GMT; Path=/; SameSite=None; Secure"),("Server","Quasar/CentralApp/Quasar/1.1.3"),("Quasar-Serv","apibeta.centralapp.com"),("Via","CentralApp/Quasar/1.1.3"),("x-powered-by","Haskell, baby."),("Access-Control-Allow-Origin","*"),("Access-Control-Allow-Credentials","true"),("Access-Control-Allow-Methods","DELETE, GET, OPTIONS, PATCH, POST, PUT"),("Access-Control-Allow-Headers","Accept, Authorization, Cache-Control, Content-Type, DNT, If-Modified-Since, Keep-Alive, Origin, User-Agent, X-Bus-URN, X-Domain, X-Mx-ReqToken, X-Requested-With")], responseBody = "{\"errors\":[{\"code\":\"ERR.INVALID_ENDPOINT\",\"message\":\"No such request is configured on this server.\",\"service\":\"Lib/GenericComp\",\"status\":404,\"type\":\"global\"}],\"services\":[\"Lib/GenericComp\"]}", responseCookieJar = CJ {expose = [Cookie {cookie_name = "AWSALBCORS", cookie_value = m"YR0tnOLtRO4hnEkt/k8O1g6hB1H567vUtd6GShkUk42UMX0ERWsScVfBPWqWQLugubSrEpMIhbwmgeo135PIOv9v6pipXF98ot34Ia0iL/sKiFyVmNTfkybUk7TM", cookie_expiry_time = 3023-01-01 00:00:00 UTC, cookie_domain = "apibeta.centralapp.com", cookie_path = "/", cookie_creation_time = 2023-08-31 08:24:39.054651166 UTC, cookie_last_access_time = 2023-08-31 08:24:39.054651166 UTC, cookie_persistent = False, cookie_host_only = True, cookie_secure_only = True, cookie_http_only = False},Cookie {cookie_name = "AWSALB", cookie_value = "YR0tnOLtRO4hnEkt/k8O1g6hB1H567vUtd6GShkUk42UMX0ERWsScVfBPWqWQLugubSrEpMIhbwmgeo135PIOv9v6pipXF98ot34Ia0iL/sKiFyVmNTfkybUk7TM", cookie_expiry_time = 3023-01-01 00:00:00 UTC, cookie_domain = "apibeta.centralapp.com", cookie_path = "/", cookie_creation_time = 2023-08-31 08:24:39.054651166 UTC, cookie_last_access_time = 2023-08-31 08:24:39.054651166 UTC, cookie_persistent = False, cookie_host_only = True, cookie_secure_only = False, cookie_http_only = False}]}, responseClose' = ResponseClose, responseOriginalRequest = Request {
host = "apibeta.centralapp.com"
port = 443
secure = True
requestHeaders = []
path = "/api/v2/no/such/path"
queryString = "?foo=bar,baz"
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
proxySecureMode = ProxySecureWithConnect
}
}
```

As you can see, `http-client` doesn't URL encode the parameters, but `wreq` seems to be doing that. Server logs also confirm this.

Is there a way to avoid this? That is, set parameters with and without URL encoding?

What I've tried so far: swapping out the default manager with one generated via `Network.HTTP.Client.TLS.newTlsManager` assuming the encoding was happening in the manager, but that has had no effect.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.