gosom / gosom/google-maps-scraper

Proxy Errors

Open
#101 13 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
5.9k
Forks
924
Avg merge
5d 13h
Merged PRs (30d)
3

Description

If I do a scrape without the -proxies flag, it works as expected.

Also if I use curl and the proxy url to the google maps url then it works as expected.

However when I add the -proxies flag I see these errors in the output and it does not work.

This is with everything installed directly on ubuntu 24.04 ... and not using Docker.

```
{"level":"info","component":"scrapemate","time":"2024-12-13T17:55:43.067587274Z","message":"starting scrapemate"}
{"level":"info","component":"scrapemate","numOfJobsCompleted":0,"numOfJobsFailed":0,"lastActivityAt":"0001-01-01T00:00:00Z","speed":"0.00 jobs/min","time":"2024-12-13T17:56:43.068652477Z","message":"scrapemate stats"}
{"level":"info","component":"scrapemate","error":"inactivity timeout: 0001-01-01T00:00:00Z","time":"2024-12-13T17:56:43.068682503Z","message":"exiting because of inactivity"}
{"level":"info","component":"scrapemate","job":"Job{ID: 71d978af-8a91-45ee-9d56-dd630099dc31, Method: GET, URL: https://www.google.com/maps/search/test, UrlParams: map[hl:en]}","error":"context canceled","status":"failed","duration":60899.47342,"time":"2024-12-13T17:56:43.967269068Z","message":"job finished"}
{"level":"error","component":"scrapemate","error":"context canceled","time":"2024-12-13T17:56:43.967304912Z","message":"error while processing job"}
{"level":"info","component":"scrapemate","time":"2024-12-13T17:56:43.967340977Z","message":"scrapemate exited"}
```

Any tips on what I could be doing wrong?

**Update1:** I tested a 2nd proxy service and I also tried setting the proxy with `export http_proxy="proxy url here"` and `export https_proxy="proxy url here"` while not using the `-proxies` flag ... but it still didn't work.

**Update 2:** I tested a 3rd proxy service using socks5 and the `-proxies` flag and got this error `{"level":"error","component":"scrapemate","error":"playwright: Browser does not support socks5 proxy authentication","time":"2024-12-14T22:00:49.630242703Z","message":"error while processing job"}`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.