dpalmasan / dpalmasan/dev-job-analytics
Investigar como automatizar scripts de ingesta de datos
- Dominant language
- JavaScript
- Stars
- 2
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
El script `backfill_so_data` es fácil de automatizar porque hace un simple request sin autorización. El script problemático es el scraper de linkedin, ya que requiere logearse desde la máquina. Idea, apuntar display a una máquina local y logearse.
```python
chrome_options = webdriver.ChromeOptions()
chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
driver = webdriver.Chrome(
executable_path=os.environ.get("CHROMEDRIVER_PATH"),
chrome_options=chrome_options,
)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the `backfill_so_data` script and the LinkedIn scraper, then inspect how each currently runs and where the Selenium Chrome setup is used. Clarify the authentication and display requirements before choosing an automation approach. Done should mean the ingestion scripts have a documented, repeatable unattended execution path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100