Builtwith API Project

### Overview
**Project:** Open Community Survey

**Volunteer Opportunity:** Create scraper to get information from builtwith.com on technologies used by neighborhood council websites. Organize the data (create categories for the tech), Automate scrape job to run periodically. Additionally, we want to display this information with a dashboard (see Google Data Studio Dashboard linked below under "Project Output" for an example).

**Contact:** ~Ryan Swan (data science), Kaylani (open community survey)~ Bonnie

### Action Items
- [x] Create a wiki page
- [x] Build a scraper that can we reuse to get the data on the NC site technologies
- [x] Add the scripts and other code to the data science repo or if another repo is required, let the leads know.
- [x] Create a Spreadsheet from the results of initial scrape
- [x] ~Create a set of categories in the spreadsheet~
- [x] Rework the script to grab category as well as technology (its available in the API)
- [x] Add category to each technology, so that the data can be grouped and analyzed will happen automatically prior item is done
- [x] Assess code for [current scraper](https://github.com/hackforla/data-science/tree/main/311-data/webscraping) to determine if it still functions properly
- [ ] Perform additional analysis on Widget technology category: Which sites are using calendars? What are the calendars used for (events of the NC or local events)? Which sites use chatbots? Which sites have search functionality? How many sites use translation widgets?
- [ ] Finish analysis of following technology categories: Content Management system (cms), Mobile, SSL, Payment, Framework, and Copyright
- [ ] Fix directory issues with code. Currently, it's in the 311 directory but needs to be moved to the open community survey directory.
- [ ] ~Create a reusable matching table of technology to category~
- [ ] ~Create a script to be able to create a new spreadsheet with the matching table so that the technologies are already categorized (except of course the ones that are new).~
- [ ] ~create instructions for updating matching table and running scripts.~
- [ ] make sure wiki is updated.
- [ ] release dependency on - https://github.com/hackforla/open-community-survey/issues/28

### Resources/Instructions

#### External Tools
- Builtwith
- https://builtwith.com
- [builtwith API](https://api.builtwith.com/)
- API limitations: Some sites, are resistant to being crawled (WordPress, for instance https://atwatervillage.org/calendar/). So what we need is a list of all the sites that can't be put through the sitemap maker. See notes about WordPress site crawling: https://community.funnelback.com/knowledge-base/implementation/Gather-And-Index/integration/crawl-wordpress-sites
- Selenium
- [Selenium Scraping Tools - see branch](https://github.com/hackforla/data-science/tree/selenium)
- Docker

#### Tutorial
- [Selenium/Docker, made by Sophia](https://drive.google.com/drive/folders/1t3AO2A8uOcKYrTGKuEzMLnsZPdPUo9Si)

### Project input (data)
- [Target Website List Here](https://docs.google.com/spreadsheets/d/1V5cOCwABGNsvp715EL-SIRklcknEJB-hj4Mtjl8zMpw/edit#gid=845441322) - this is one tab on a larger analysis workbook.

### Project output
- [ Data Science wiki, 99 NC project](https://github.com/hackforla/data-science/wiki/99-Neighborhood-Council-Websites-Technologies-Used-Analysis)
- [Spreadsheet of Rajinder's script results](https://docs.google.com/spreadsheets/d/1gvlBd-tragQmT1oblX560-EWbDXYzrcqVDUhyIzHWa0/edit#gid=1851194011)
- [Spreadsheet of updated script results](https://docs.google.com/spreadsheets/d/1IDpVrv6kr06nvbr3nI09933gFdePukhvB5J39XruwAA/edit#gid=151164025)
- [Example of Google Data Studio Dashboard](https://datastudio.google.com/u/0/reporting/98d7a4a7-9092-44f9-937f-cd3cd2a9cd27/page/p_02cc3mo7uc)
#### Rajinder's code
- [code on data-science repo with Rajinder + Willa's code](https://github.com/hackforla/data-science/tree/main/311-data/webscraping) - this will need to be moved to another directory. It has nothing to do with 311. Its a project for Open Community Survey.
- [Rajinder's personal repo](https://github.com/rajindermavi/Webscraping) - this seems to be updated more recently than the one on data-science.

### Current presentation
[OCS: Tech usage insights NCs](https://docs.google.com/presentation/d/1BOPJM_m3BYzC3XSq2rKtlv3kj64FWjU4hq7yeyl7pcg/edit#slide=id.g15af70a5a28_0_14)
[Analytics Analysis Workbook](https://docs.google.com/spreadsheets/d/1ddoq6qx_EbZ5yWvkMXdOYPh_IO68PcohR1xMeR3EseA/edit#gid=389805109)
[Widgets Analysis Workbook](https://docs.google.com/spreadsheets/d/1pUVd5nEicyKnELFL8k4NJJVdnfbBNMwo/edit?usp=sharing&ouid=110074812176014669614&rtpof=true&sd=true)

#### Related issues from OCS
- https://github.com/hackforla/open-community-survey/issues/25

---
**Past Collaborators:**
```@akibrhast, @ava li, @Sarah Williams, @wendywilhelm10 @rajindermavi @ShikaZzz @JessicaFB @Poorvi Rao```