refactor(owasp): Improve error handling in chapter scraping command
- Dominant language
- Python
- Stars
- 451
- Forks
- 707
- Avg merge
- 22h 59m
- Merged PRs (30d)
- 91
Description
**Is your feature request related to a problem? Please describe.**
The management command `owasp_scrape_chapters` relies on the scraper functions in `backend/apps/owasp/scraper.py`. Web scraping scripts are inherently fragile and can break if the HTML structure of the target website (owasp.org) changes.
Currently, if an element is not found during scraping, the command might fail with a generic error (e.g., `AttributeError: 'NoneType' object has no attribute 'get'`) or fail silently, leading to incomplete or missing chapter data in the database. This makes it difficult to diagnose and fix scraping issues when they arise.
**Describe the solution you'd like**
I propose to refactor the chapter scraping logic to make it more resilient and provide better diagnostics. This would involve:
1. **Adding Robust Error Handling:** Wrap the scraping logic for individual data points (like finding a chapter's name, region, or leaders) in `try...except` blocks.
2. **Implementing Detailed Logging:** When an element cannot be found or an error occurs for a specific chapter, log a detailed warning message. The message should include which chapter was being processed and which specific piece of information was missing.
**Describe alternatives you've considered**
An alternative is to have the script fail immediately on any error. However, this is less ideal because a minor change on the source website for a single chapter could prevent all other chapters from being updated. A more resilient approach with good logging is preferable.
**Are you going to work on implementing this?**
- [x] Yes
- [ ] No
**Additional context**
* **Files to be modified:**
* `backend/apps/owasp/scraper.py` (to implement the core logic)
* `backend/apps/owasp/management/commands/owasp_scrape_chapters.py` (to ensure it calls the updated scraper functions correctly)
**Assignee Request:**
I would like to implement this feature. Please assign this issue to me.
(Hacktoberfest)
Contributor guide
Research direction
Read backend/apps/owasp/scraper.py and backend/apps/owasp/management/commands/owasp_scrape_chapters.py, then run the owasp_scrape_chapters management command to observe current failures. The work is done when missing chapter data produces detailed warnings identifying the chapter and field, while errors affecting one chapter do not prevent other chapters from being processed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- django, python
- Domain
- backend, cli
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100