sillsdev / sillsdev/languageforge-lexbox

Optimize Component and Complex form creation/validation

Open
#2,105 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

📖 MiniLcm
Dominant language
C#
Stars
9
Forks
8
Avg merge
2d 13h
Merged PRs (30d)
49

Description

Describe the feature

The Crdt version of the Test CanSyncRandomEntry takes significantly longer than the fwdata version.

Here's the profiling data of the crdt version of the test with no round-trip API, which seems to confirm our suspicion that complex forms and components operations are quite expensive due to preventing reference cycles.
Profiling data was collected in Visual Studio by putting break points at the beginning and end of the test and using Record CPU Profile in the Diagnostic Tools view.
(Sadly I could not find a way to export any of the profiling results, but I've included lots of screenshots)

TLDR;
Checking for reference cycles or duplicate complex-forms/components accounts for approximately 1/5th of the entry sync time.

Summary of the test:
  1. We create an entry with 5 of everything (senses, example-sentences, complex-forms, components etc.)
  2. We make a significantly altered version of that entry:
  • 5 new things of everything
  • heavy modifications and deletions of everything

So:

  • 5 components and 5 complex-forms are added
  • components are moved around and some are deleted
  • 5 senses and 25 example sentences are added
  • existing senses and example sentences are shuffled and some are deleted
Bird's eye view

For profiling We focus only on the EntrySync.SyncFull(before, after, Api) code, because everything else is test infrastructure. (e.g. PrepareToCreateEntry).

SyncComplexFormsAndComponents and SyncWithoutComplexFormsAndComponents are almost equally expensive.
One would think that the former is significantly cheaper, because it only syncs 2 fields, but it's actually more expensive than syncing the entries and their children.

Image
Sync without components

Senses are by far the most expensive step. That makes sense, because they're a tree of entities:

Image
Senses

Time syncing senses is fairly evenly divided between: Add, Replace and Remove:
Image

  • Add: No specific bottleneck
Image
  • Replace: all the time is spend on example-sentences, because that's all we change on existing senses:
Image
  • Remove: nothing noteworthy
Image
Sync with components

Almost all the time is spent on Components rather than Complex Forms:
(is that maybe simply, because we sync Components first? They both have to check for reference cycles.)

Image
Components

Adding was by far the most expensive.
Looks like nothing was reordered, which is unexpected.

Image
  • Adding: basically all of the time is spent checking for reference cycles or duplicates in the database:
Image
Flamegraph overview
Image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with EntrySync.SyncFull(before, after, Api), comparing SyncComplexFormsAndComponents and SyncWithoutComplexFormsAndComponents, then use the CanSyncRandomEntry test and the profiling screenshots to locate the cycle or duplicate checks. Done means reducing the cost of component and complex-form synchronization while preserving the test's sync behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.