hypothesis / hypothesis/product-backlog
Get rid of the `extra` annotation field
- Dominant language
- No language data
- Stars
- 122
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
## Background
Since we switched our primary annotation store from Elasticsearch to Postgres, we’ve had an `extra` field on our annotation model that stores any data from API requests that the service doesn’t understand; this is then presented back out into API responses.
Since our work on moderation, we’ve introduced keys that pertain to the particular user making the request (say, whether they’ve flagged an annotation), or are only visible to certain users (aggregate flag information). We’ve had to add [specific logic to filter these fields out](https://github.com/hypothesis/h/blob/master@%7B2017-04-10%7D/h/schemas/annotation.py#L265-L275) and stop them being persisted to Postgres.
@nickstenning has done a bit of spelunking into the data and has come back with the most common keys in our current data:
Key|Count
---|-----
consumer|104785
links|7526
ranges|4695
tag|3004
selector|413
highlightText|163
flagged|107
nipsa|79
smokey|65
title|48
source_stripped|48
source|48
favicon_link|48
gsx$user|18
gsx$uri|18
gsx$updated|18
gsx$text|18
gsx$target0source|18
gsx$target0selector3type|18
gsx$target0selector3suffix|18
Our current null hypothesis is that the contents of this field are **100% junk**, and in some cases potentially harmful (for another example, the `nipsa` field above, which might potentially be exposed to everyone, including the NIPSA’d user).
Given the lack of known benefits, and the presence of known costs, to keeping this field around, our current suggestion is to remove it entirely.
## Questions
* What should we do with fields we don’t understand? Should we ignore them, HTTP/HTML/CSS style, or raise an error?
* Is it worth testing our hypothesis that this is all junk data? For instance, not including it in API responses and seeing whether anyone complains before we drop it from the database entirely?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with h/schemas/annotation.py, especially the linked filtering logic, then trace how annotation API requests and responses interact with Postgres persistence. The issue needs a decision on handling unknown fields and whether to remove extra from responses and the database; it does not define a concrete done state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql
- Domain
- api, backend, databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100