JustAnotherArchivist / JustAnotherArchivist/snscrape

Add a JSON Schema

Open
#611 2 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
5.4k
Forks
783
PR merge metrics
No merged PRs in 30d

Description

The JSON currently produced by snscrape has some issues: deprecated fields[^0] can't be marked as such (compatibility), and default values are included (bloat). This can't be fixed within JSON, but JSON Schema can solve these problems. To avoid repeating information across the repo, the schema should be generated from the dataclasses. [GenSON](https://pypi.org/project/genson/) appears to be the primary tool for things like that, though it doesn't support the `deprecated` property, so at least some manipulation of the schema would be required. It also can't use the dataclasses with their type annotations directly.

[^0]: Deprecated fields are currently not included in the generated JSON at all, and marking them after the soon-to-come addition is what prompted me to think about this in the first place.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names dataclasses and the generated JSON but no files or tests. Start by locating the dataclasses and the JSON serialization path, then review how deprecated fields and default values are represented. Done means a JSON Schema is generated from the dataclasses and represents those fields without duplicating their definitions across the repository.

Written by the indexing model from the issue text.

Assessment

Tech stack
json, python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.