Morango memory performance improvements

What problem are we solving?

• Memory consumption performance issues with large syncs
• Addressing architectural things that will further enhance “streaming” syncs

How do we know this is successful?

• When a large instance (VF DRC on BCK) can sync with KDP, without getting OOM killed on either side

Who is this for?

• Our own sanity
• Our users and timely access to their facility data

Are there any assumptions or constraints?

• We’re likely not going to make the sync any faster by addressing memory issues
  • Although, with proactive planning, we can lay out the improvements to later on address the loss of speed with more drastic restructuring of the sync process itself
• We should ensure we have locking mechanisms in place to prevent concurrent syncing complications

What approaches/risks were discussed during planning?

• Overall, breaking down the syncing process into ETL (extract, transform, load) procedures
  • Source models →Transform → Store
  • Store → Transform → Buffer
  • Buffer → Transfer → Buffer
  • Buffer → Transform → Store
  • Store → Transform → Source models
• Morango doesn’t differentiate between single field updates and full record updates
• Feasibility of progress tracking in queuing, serialization, dequeuing, and deserialization
• How the data structure or handling will change with other planned work— possibly integrating CRDTs
• Managing foreign keys and data relationships to ensure data integrity

What is out of scope?

• Addressing slow queries
• Directly addressing speed performance loss

Acceptance Criteria

• Backwards compatibility
• Improved memory performance for both Postgres and SQLite database backends
• Architectural improvements that will enable us to scale it further

Workspace & Resources

• https://gist.github.com/bjester/c2a987e484c9fa34fc344c81f4f19f4e