KaykCaputo / KaykCaputo/SisuDataQuery

Replace Selenium with Direct HTTP Requests

Open
#1 0 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
C#
Stars
3
Forks
0
PR merge metrics
No merged PRs in 30d

Description

**Description:**

The project currently uses Selenium WebDriver to fetch data from URLs such as:
https://sisu-api.sisu.mec.gov.br/api/v1/oferta/instituicoes/uf
https://sisu-api.sisu.mec.gov.br/api/v1/oferta/instituicao/{iesCode}
[https://sisu-api.sisu.mec.gov.br/api/v1/oferta/{offerCode}/modalidades](https://sisu-api.sisu.mec.gov.br/api/v1/oferta/%7BofferCode%7D/modalidades)

These endpoints return **raw JSON**, so using Selenium to load a browser, parse HTML, and extract the JSON from a `

` tag is unnecessary. This makes the program slower, more fragile, and more resource-heavy.  

This issue proposes replacing all Selenium-based HTTP fetching with direct `HttpClient` GET requests.

---

### **Tasks**

- [ ] Create a static `HttpClient` instance, e.g.:
`private static readonly HttpClient client = new HttpClient();`
- [ ] Replace all `driver.Navigate().GoToUrl(url)` calls with:
`var json = await client.GetStringAsync(url);`
- [ ] Remove HTML parsing using HtmlAgilityPack
- [ ] Deserialize JSON directly using `JsonConvert.DeserializeObject`
- [ ] Preserve existing data structures (`Institution`, `Course`, `CourseWeights`, etc.)
- [ ] Remove Selenium and HtmlAgilityPack from the project dependencies
- [ ] Update README to reflect that Selenium is no longer required
- [ ] Ensure all current features behave identically after the change

---

### **Why This Is Good for Beginners**

- Involves simple and clear modifications
- Improves code performance and maintainability
- No advanced browser automation required
- Easy to test because the APIs are public and return consistent JSON
- Gives contributors experience with networking and JSON parsing

---

### **Optional Enhancements**

- [ ] Add error handling for network failures
- [ ] Implement timeout for requests
- [ ] Validate missing or malformed JSON fields

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.