OWASP / OWASP/Nest

refactor(owasp): Improve error handling in chapter scraping command

Open
#2,439 7 comments 0 reactions 0 assignees View on GitHub
bug enhancement
Dominant language
Python
Stars
451
Forks
707
Avg merge
22h 59m
Merged PRs (30d)
91

Description

**Is your feature request related to a problem? Please describe.**
The management command `owasp_scrape_chapters` relies on the scraper functions in `backend/apps/owasp/scraper.py`. Web scraping scripts are inherently fragile and can break if the HTML structure of the target website (owasp.org) changes.

Currently, if an element is not found during scraping, the command might fail with a generic error (e.g., `AttributeError: 'NoneType' object has no attribute 'get'`) or fail silently, leading to incomplete or missing chapter data in the database. This makes it difficult to diagnose and fix scraping issues when they arise.

**Describe the solution you'd like**
I propose to refactor the chapter scraping logic to make it more resilient and provide better diagnostics. This would involve:

1. **Adding Robust Error Handling:** Wrap the scraping logic for individual data points (like finding a chapter's name, region, or leaders) in `try...except` blocks.
2. **Implementing Detailed Logging:** When an element cannot be found or an error occurs for a specific chapter, log a detailed warning message. The message should include which chapter was being processed and which specific piece of information was missing.

**Describe alternatives you've considered**
An alternative is to have the script fail immediately on any error. However, this is less ideal because a minor change on the source website for a single chapter could prevent all other chapters from being updated. A more resilient approach with good logging is preferable.

**Are you going to work on implementing this?**

- [x] Yes
- [ ] No

**Additional context**
* **Files to be modified:**
* `backend/apps/owasp/scraper.py` (to implement the core logic)
* `backend/apps/owasp/management/commands/owasp_scrape_chapters.py` (to ensure it calls the updated scraper functions correctly)

**Assignee Request:**
I would like to implement this feature. Please assign this issue to me.

(Hacktoberfest)

Contributor guide

Open the contributing guide

Research direction

Read backend/apps/owasp/scraper.py and backend/apps/owasp/management/commands/owasp_scrape_chapters.py, then run the owasp_scrape_chapters management command to observe current failures. The work is done when missing chapter data produces detailed warnings identifying the chapter and field, while errors affecting one chapter do not prevent other chapters from being processed.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
backend, cli
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.