0surface / 0surface/arxiveprior

Redesign PublicationItemEntity Table Storage Class

Open
#25 0 comments 0 reactions 1 assignee Claimed by @0surface View on GitHub
Dominant language
C#
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

[Not Required] Redesign to handle large Author collections don't exceed the Azure Table Storage's property size limit of 64 Kb

Implemented Solution
From PublicationItem -> PublicationItemEntity
Added custom Auto Mapper resolver (AuthorListSplitResolver) to split excess Author data into 3 more Author list properties, in chunks of 400.
That is 4 List of AuthorItem properties each holding 400 items.
Any left will not be persisted,and the '_AuthorListTruncated_' flag is set to true.

From PublicationItemEntity -> PublicationItem
Added Auto Mapper resolver (AuthorListAggregateResolver) to add any authors in the 'spill' over Author list properties back to te single Authors property of the PublicationItem type.

See arxiv id 2007.06722v3 which has 948 Authors i.e. (name, affiliation) pair in JSON format.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue involves redesigning the PublicationItemEntity table storage class to handle large author collections within Azure Table Storage's 64 KB property limit. Look at the existing PublicationItem to PublicationItemEntity mapping and the custom AutoMapper resolvers (AuthorListSplitResolver and AuthorListAggregateResolver). Examine how authors are split into chunks of 400 across multiple properties and how the '_AuthorListTruncated_' flag is used. Test with a paper like arXiv ID 2007.06722v3 that has 948 authors to verify the implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.