Socks proxy isn't used (by the looks of things)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 4.1k
- Forks
- 673
- PR merge metrics
- No merged PRs in 30d
Description
-
Ripme version:1.7.92
-
Java version: openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-3ubuntu1)
OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-3ubuntu1, mixed mode, sharing) -
Operating system: Ubuntu 20.04 LTS x86_64
-
Exact URL you were trying to rip when the problem occurred: Any URL which is blocked without my proxy.
-
Please include any additional information about how to reproduce the problem: Running tor proxy locally.
This is getting in the way of two new ripper PRs I have that I think work (but can't test)
Expected Behavior
Use socks proxy provided in the config (proxy.socks = 127.0.0.1:9050). Working for loading the same urls in a browser. I also tried a few other random proxies from the internet and they didn't work either.
Actual Behavior
Always fails to load after 1 attempt (Same as without a proxy in the config)
java.io.IOException: Failed to load https://xhamster.com/videos/brazzers-busty-big-booty-milf-lisa-ann-fucks-her-masseur-1492828 after 1 attempts
at com.rarchives.ripme.utils.Http.response(Http.java:200)
at com.rarchives.ripme.utils.Http.get(Http.java:159)
at com.rarchives.ripme.ripper.rippers.XhamsterRipper.getFirstPage(XhamsterRipper.java:111)
at com.rarchives.ripme.ripper.AbstractHTMLRipper.rip(AbstractHTMLRipper.java:94)
at com.rarchives.ripme.ripper.AbstractRipper.run(AbstractRipper.java:616)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: Connect timed out
at java.base/java.net.SocksSocketImpl.readSocksReply(SocksSocketImpl.java:138)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:509)
at java.base/java.net.Socket.connect(Socket.java:609)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:434)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:181)
at com.rarchives.ripme.utils.Http.response(Http.java:179)
... 5 more
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in com.rarchives.ripme.utils.Http at the response, get, and failure paths named in the stack trace, then check how proxy.socks is passed into requests. Reproduce with the local SOCKS proxy at 127.0.0.1:9050 and a blocked URL such as the reported Xhamster URL. Done means the configured proxy is used and the URL loads successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100