IQSS / IQSS/dataverse

SPIKE: Come up with a new design for storing access information, that does not rely on an infinitely growing GuestBookResponse table. (via an archiving mechanism?)

Open
#12,618 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature: Guestbook Feature: Performance & Stability Size: 20
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

We need to re-engineer how we store, and count downloads and other access calls.
Close to 1/3 of our prod. db appears to be the guestbookresponse table now.

Unlike the actionlogrecord table on which we perform a yearly bloodletting, there is no solution for archiving guestbookresponse under the current setup, since we rely on counting the entries in the table for download counts on files and datasets, for display and/or metrics; and because all the downloads and guestbooks data are supposed to be available for viewing and downloading.

I am assuming that it should be doable to add an archiving mechanism. That could be used to delete N years-worth of the oldest records; while preserving the raw counts for display and metrics. And would provide some form of access to them once they are archived. For example, it can be communicated to the users that only the last M years of access records are easily accessible, and the older records need to be retrieved from S3, perhaps 1 year at a time.

I have some ideas that I will add. But I am encouraging everybody to participate/brainstorm.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how GuestBookResponse records are stored and counted for downloads, display, and metrics, then compare that with the existing ActionLogRecord archiving approach. Done means an agreed design for reducing table growth while preserving raw counts and a defined way to retrieve older access records.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.