hypothesis / hypothesis/product-backlog

Get rid of the `extra` annotation field

Open
#265 2 comments 0 reactions 0 assignees View on GitHub
Design
Dominant language
No language data
Stars
122
Forks
7
PR merge metrics
No merged PRs in 30d

Description

## Background

Since we switched our primary annotation store from Elasticsearch to Postgres, we’ve had an `extra` field on our annotation model that stores any data from API requests that the service doesn’t understand; this is then presented back out into API responses.

Since our work on moderation, we’ve introduced keys that pertain to the particular user making the request (say, whether they’ve flagged an annotation), or are only visible to certain users (aggregate flag information). We’ve had to add [specific logic to filter these fields out](https://github.com/hypothesis/h/blob/master@%7B2017-04-10%7D/h/schemas/annotation.py#L265-L275) and stop them being persisted to Postgres.

@nickstenning has done a bit of spelunking into the data and has come back with the most common keys in our current data:

Key|Count
---|-----
consumer|104785
links|7526
ranges|4695
tag|3004
selector|413
highlightText|163
flagged|107
nipsa|79
smokey|65
title|48
source_stripped|48
source|48
favicon_link|48
gsx$user|18
gsx$uri|18
gsx$updated|18
gsx$text|18
gsx$target0source|18
gsx$target0selector3type|18
gsx$target0selector3suffix|18

Our current null hypothesis is that the contents of this field are **100% junk**, and in some cases potentially harmful (for another example, the `nipsa` field above, which might potentially be exposed to everyone, including the NIPSA’d user).

Given the lack of known benefits, and the presence of known costs, to keeping this field around, our current suggestion is to remove it entirely.

## Questions

* What should we do with fields we don’t understand? Should we ignore them, HTTP/HTML/CSS style, or raise an error?
* Is it worth testing our hypothesis that this is all junk data? For instance, not including it in API responses and seeing whether anyone complains before we drop it from the database entirely?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with h/schemas/annotation.py, especially the linked filtering logic, then trace how annotation API requests and responses interact with Postgres persistence. The issue needs a decision on handling unknown fields and whether to remove extra from responses and the database; it does not define a concrete done state.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
api, backend, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.