CodeForPhilly / CodeForPhilly/MATchMapper

Script to convert scraped data (in csv or pandas df) to json format for Django

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Write a script that works with the output from the scraper (could be directly with .csv or the pandas df) and converts the output into json data that can be sent to the Django API to input into our database. ** Important: If you are taking this issue, make sure you make clear documentation as to whether you have decided to work directly with the pandas df or the csv format output from the scrapers. It would be best if you make a decision and let the rest of the team know. After making the decision, you should also reconfigure the scraping files such that they become functions with a boolean parameter that allows the user to decide if he or she wants the scraping scripts to output a .csv file or not.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the scraper files and tracing whether their output is a pandas DataFrame or CSV, then inspect the Django API input path and database expectations. Decide and document one input format, add the conversion to JSON, and reconfigure the scraping files to expose the requested CSV-output option.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, pandas, python
Domain
api, backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.