dpalmasan / dpalmasan/dev-job-analytics

Investigar como automatizar scripts de ingesta de datos

Open
#64 0 comments 0 reactions 0 assignees View on GitHub
scraper
Dominant language
JavaScript
Stars
2
Forks
1
PR merge metrics
No merged PRs in 30d

Description

El script `backfill_so_data` es fácil de automatizar porque hace un simple request sin autorización. El script problemático es el scraper de linkedin, ya que requiere logearse desde la máquina. Idea, apuntar display a una máquina local y logearse.

```python
chrome_options = webdriver.ChromeOptions()
chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
driver = webdriver.Chrome(
executable_path=os.environ.get("CHROMEDRIVER_PATH"),
chrome_options=chrome_options,
)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the `backfill_so_data` script and the LinkedIn scraper, then inspect how each currently runs and where the Selenium Chrome setup is used. Clarify the authentication and display requirements before choosing an automation approach. Done should mean the ingestion scripts have a documented, repeatable unattended execution path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.