dakrone / dakrone/clj-http

Compatibility issue with get request

Open
#540 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Clojure
Stars
1.8k
Forks
410
No merged PRs in 30d

Description

When I make a request like so

(http/get "https://angel.co")

I get the response

1. Unhandled clojure.lang.ExceptionInfo
   clj-http: status 403
   {:cached nil,
    :request-time 141,
    :repeatable? false,
    :protocol-version {:name "HTTP", :major 1, :minor 1},
    :streaming? true,
    :http-client
    #object[org.apache.http.impl.client.InternalHttpClient 0x7d5d53f8 "org.apache.http.impl.client.InternalHttpClient@7d5d53f8"],
    :chunked? true,
    :type :clj-http.client/unexceptional-status,
    :reason-phrase "Forbidden",
    :headers
    {"Server" "cloudflare",
     "Content-Type" "text/html; charset=UTF-8",
     "X-Frame-Options" "SAMEORIGIN",
     "Connection" "close",
     "cf-request-id" "02afa1c33a00000cb192b11200000001",
     "Transfer-Encoding" "chunked",
     "Set-Cookie"
     "__cfduid=d5186984dc436aac8989538666ddb21761589373348; expires=Fri, 12-Jun-20 12:35:48 GMT; path=/; domain=.angel.co; HttpOnly; SameSite=Lax",
     "Expect-CT"
     "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
     "CF-RAY" "592c6be52bb00cb1-EWR",
     "Date" "Wed, 13 May 2020 12:35:48 GMT",
     "Vary" "Accept-Encoding",
     "Cache-Control" "no-cache",
     "CF-Chl-Bypass" "1"},
    :orig-content-encoding nil,
    :status 403,
    :length -1,
    :body
    "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]>    <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Attention Required! | Cloudflare</title>\n<meta name=\"captcha-bypass\" id=\"captcha-bypass\" />\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge,chrome=1\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" type=\"text/css\" media=\"screen,projection\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" type=\"text/css\" media=\"screen,projection\" /><![endif]-->\n<style type=\"text/css\">body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!--><script type=\"text/javascript\" src=\"/cdn-cgi/scripts/zepto.min.js\"></script><!--<![endif]-->\n<!--[if gte IE 10]><!--><script type=\"text/javascript\" src=\"/cdn-cgi/scripts/cf.common.js\"></script><!--<![endif]-->\n\n\n\n\n</head>\n<body>\n  <div id=\"cf-wrapper\">\n    <div class=\"cf-alert cf-alert-error cf-cookie-error\" id=\"cookie-alert\" data-translate=\"enable_cookies\">Please enable cookies.</div>\n    <div id=\"cf-error-details\" class=\"cf-error-details-wrapper\">\n      <div class=\"cf-wrapper cf-header cf-error-overview\">\n        <h1 data-translate=\"challenge_headline\">One more step</h1>\n        <h2 class=\"cf-subheadline\"><span data-translate=\"complete_sec_check\">Please complete the security check to access</span> angel.co</h2>\n      </div><!-- /.header -->\n      \n      <div class=\"cf-section cf-highlight cf-captcha-container\">\n        <div class=\"cf-wrapper\">\n          <div class=\"cf-columns two\">\n            <div class=\"cf-column\">\n            \n              <div class=\"cf-highlight-inverse cf-form-stacked\">\n                <form class=\"challenge-form\" id=\"challenge-form\" action=\"/?__cf_chl_captcha_tk__=0d94e2e21d2ef06ef34bd2b5b4667f279b690108-1589373348-0-ATT4PiY_pQI1dGVw0_sZDV32_7x4mqtO4RepyD-L4i6zBJiIuml25fVlyJaK8uXNJw5ZWnzGlb6y0jGJJ8HIdEz14sOXRUoHqs_naHtwFEQywa8qZf_rwHsBxIUD5y_FNPph6TDrcfLVnQaN9eyy5VjiznzH4y0yeK8cidNnd-qNGw4OIZbFLfv8299DGhvNnBgsbn3BiQ9bkoGOtE4wANUh5U2LTJVAWhlquAvfhjCu6jHlYRXtN5GdnNvfBbCYwWGwCX0j88J-qCjJFOrSvx1_xraYtpB_Y8PpLHZTob_t8POfE0kJpn9ZYxwjhLQhqAAcIoE8fRe7Lv_50pzummklgMLgTRT2_NJGiE-_jNEogQmoTCvGOOmhNCe28SVYkXop9Ajm-z-6xwgoKQnY7EwekXJZCs-4nwpWJ9Gh3HBgVxZRiuv_wKgcmU0sPlLXSL5G8yOVdbBKBtHhQyqadtmTSg_IC2HV7SiYqPoJMmpJGfxxUm1au7ZS9ZiLpokjI5pQDZLpT2ZG-6jVfnTKvt9w_qmMtUSBhDleXd8mG59r\" method=\"POST\" enctype=\"application/x-www-form-urlencoded\">\n  <input type=\"hidden\" name=\"r\" value=\"bfe8db4864c274e3ed80528a0e0ad233279c00b9-1589373348-0-AVWaRjujNq/XSmYrRyYxyBLhp5bbxA92rBX2qiiOx9PVWzas1b/usxApmblw248v1q5iUvP/V/GYHXhQF1UBviAqExhVjGW4upmNgdEf/zdFWHbgQb/s0RdZyMS+rurne8Y7aKD8ppx/WHjY8eSxVTGcHePc+qs/NdCt33voCLk2sGd0inuxibNjFXkBT62qs/JshlzaDsM58mC/jdSBRHZiOoJHmteJ0J1vwDVTVumWM97Qrc9fDyAqvDo72LCfqq0uG6hppWsi/z5jnGhTwzmJ7biqcY3BThvQAABSgD80MH4unfjys3iYhsefX0tfuAm23Rx1BCoKDRrrnWy0//Z9D0vI3petRmLSerLnJUAqCRh6ZoRqahYwNTPr39G+/WBJBsh3UDfB0+PwSmGsczRmL6DDbDu023etpAhehWcdR55ftEcijKiEnnfZE4vyKYm4C835QoKlQ+odT+u7syO/u/PgoyguQxqnNoKdlSSCs4+96s86urmY/yM9T4dvZdB4K4aOVH5cNfRHc8fsqeKpcuxmBbHOmIYIAegjTd5iKB4OQtxPHti1ZQCLeP74OiAxF6UgH+bCBp+h2mfU19CtEXvfcQdxGXPDT/iAPbPZG8c7fubDCKUympyb5nbHzVUcL9IGTlCq1zN7B1pRFj/O6JKOGBRo+q0OEs0nI7l/RFvmDfEtA0FYSC4IGegEs//fUsB165Zdm2SdKk7/cy89Xd4Hy5cedzqmjrtKNw5zjvfjqaNU7FlUL38irfopK/Pyk5Fp/HdV7iMvflIJO1M7GedTWdcNKB/OqPGV9NuJaKYgJbgBrxS4iYtHw9ZZsKWogYCig+eYiU8ty/MSDus9zCE2yRIbLVQ59AFwqTwODgBaV2nJepBDxcXVauCpdHiGbi7Q9M4t1eyGafFUKasv3unzdriRTrFPZ+44ZQb3gYberTMv2f3MwfcryaFgxcgtu43w8Hy5nviA9sOeoLmPYMZtL85QbB+AzKCXJV5DfIGcMvx1aeD/D9QNyOSTakVv2tAwxnP5UeQj8mJKGHTYrIsOMFDfxSnQ2lVzMRPQYmeEes8KjFvYrGyQ82Io+hGnKYOHX1T1ioi+wh+MGacVaSC1VMfG6rdIauPSxbB9WNxqnJxKz7SxHNiV3Gwm4rgUOs+vN2tSPyfINt12OHU=\">\n  <input type=\"hidden\" name=\"cf_captcha_kind\" value=\"h\">\n  <script type=\"text/javascript\" src=\"/cdn-cgi/scripts/hcaptcha.challenge.js\" data-type=\"normal\"  data-ray=\"592c6be52bb00cb1\" async data-sitekey=\"33f96e6a-38cd-421b-bb68-7806e1764460\"></script>\n  <noscript id=\"cf-captcha-bookmark\" class=\"cf-captcha-info\">\n  <h1 data-translate=\"turn_on_js\" style=\"color:#bd2426;\">Please turn JavaScript on and reload the page.</h1>\n  </noscript>\n  <div id=\"trk_captcha_js\" style=\"background-image:url('/cdn-cgi/images/trace/captcha/nojs/h/transparent.gif?ray=592c6be52bb00cb1')\"></div>\n</form>\n\n              </div>\n            </div>\n\n            <div class=\"cf-column\">\n              <div class=\"cf-screenshot-container\">\n              \n                <span class=\"cf-no-screenshot\"></span>\n              \n              </div>\n            </div>\n          </div><!-- /.columns -->\n        </div>\n      </div><!-- /.captcha-container -->\n\n      <div class=\"cf-section cf-wrapper\">\n        <div class=\"cf-columns two\">\n          <div class=\"cf-column\">\n            <h2 data-translate=\"why_captcha_headline\">Why do I have to complete a CAPTCHA?</h2>\n            \n            <p data-translate=\"why_captcha_detail\">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>\n          </div>\n\n          <div class=\"cf-column\">\n            <h2 data-translate=\"resolve_captcha_headline\">What can I do to prevent this in the future?</h2>\n            \n\n            <p data-translate=\"resolve_captcha_antivirus\">If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.</p>\n\n            <p data-translate=\"resolve_captcha_network\">If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.</p>\n            \n              \n            \n          </div>\n        </div>\n      </div><!-- /.section -->\n      \n\n      <div class=\"cf-error-footer cf-wrapper\">\n  <p>\n    <span class=\"cf-footer-item\">Cloudflare Ray ID: <strong>592c6be52bb00cb1</strong></span>\n    <span class=\"cf-footer-separator\">&bull;</span>\n    <span class=\"cf-footer-item\"><span>Your IP</span>: 128.151.150.1</span>\n    <span class=\"cf-footer-separator\">&bull;</span>\n    <span class=\"cf-footer-item\"><span>Performance &amp; security by</span> <a href=\"https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer\" id=\"brand_link\" target=\"_blank\">Cloudflare</a></span>\n    \n  </p>\n</div><!-- /.error-footer -->\n\n\n    </div><!-- /#cf-error-details -->\n  </div><!-- /#cf-wrapper -->\n\n  <script type=\"text/javascript\">\n  window._cf_translation = {};\n  \n  \n</script>\n\n\n</body>\n</html>\n",
    :trace-redirects []}
               support.clj:  201  slingshot.support/stack-trace
                client.clj:  253  clj-http.client/exceptions-response
                client.clj:  244  clj-http.client/exceptions-response
                client.clj:  262  clj-http.client/wrap-exceptions/fn
                client.clj:  734  clj-http.client/wrap-accept/fn
                client.clj:  756  clj-http.client/wrap-accept-encoding/fn
                client.clj:  717  clj-http.client/wrap-content-type/fn
                client.clj:  958  clj-http.client/wrap-form-params/fn
                client.clj:  992  clj-http.client/wrap-nested-params/fn
                client.clj: 1016  clj-http.client/wrap-flatten-nested-params/fn
                client.clj:  892  clj-http.client/wrap-method/fn
               cookies.clj:  131  clj-http.cookies/wrap-cookies/fn
                 links.clj:   63  clj-http.links/wrap-links/fn
                client.clj: 1045  clj-http.client/wrap-unknown-host/fn
                client.clj: 1173  clj-http.client/request*
                client.clj: 1166  clj-http.client/request*
                client.clj: 1179  clj-http.client/get
                client.clj: 1175  clj-http.client/get
               RestFn.java:  410  clojure.lang.RestFn/invoke
                      REPL:   62  user/eval44577
                      REPL:   62  user/eval44577
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   91  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   18  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  155  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  190  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  189  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  748  java.lang.Thread/run

Where the body says <html class="no-js ie6 oldie" lang="en-US">. How do I fix this? It's supposed to return the actual webpage?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with clj-http.client/get and the reported stack-trace paths in client.clj, especially request* and exceptions-response. Reproduce the request to angel.co and inspect whether the 403 response is caused by clj-http or the site's Cloudflare challenge. Done when the compatibility behavior and any required client change are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.