bcgov / bcgov/entity

NAMES GCP: Migrate NameX Solr Instance

Open
#26,166 0 comments 0 reactions 3 assignees Assigned to @kialj876 View on GitHub
Entities - Data Migration GCP Migration Names Team P SOLR SRE techdebt
Dominant language
JavaScript
Stars
23
Forks
62
Avg merge
24m
Merged PRs (30d)
1

Description

The NameX Solr instance in OCP is moving to GCP.

Also, the Solr instance in OCP is open to the public without authentication so is in need of a security upgrade.

The point of this ticket is to:
- [ ] Make Solr more secure
- [ ] Migrate Solr Instance from OCP to GCP across all environments - including data migration, Dockerfile & CI/CD updates, and potentially Solr version upgrades

### Overall Plan:
- Mirror the BOR architecture
- user managed leader / follower
- will have a leader + follower image running on vm instances behind their own load balancers
- wrap with a small api + same solr service for serving searches, updates, ops operations, etc.
- will require small changes in namex api to talk to new solr api instead of solr directly
- similar managed synonym approach
- will require small change in solr-admin-app to keep it updated
- stemming is applied during indexing: synonym list should not contain multiple words with the same stem (Future: could solr-api could check for this before adding it)
- scheduled job for reindexing regularly
- Replace 3 cores with 1 NR core
- will have a nested schema for the names
- will require small updates to the solr update streams
- could drastically improve the names examiner conflicts list by using something similar to the bor solr alg (potentially allowing for auto approval for many cases)
- for now planning to simulate the same functionality as the current namex
- will require updates to the current solr queries used by NameX
- will be useful for other things like BRD
- easy to expand for other use cases (i.e. search table in examiner app)
- loading / reindexing will not require reaching into postgres wrapper / oracle db, and can simply pull from the namex db
- will need regression testing NRs from draft through all possible scenarios

### Future

NameX
- Training NameX team to understand the solr api wrapper (this is much easier to learn than everything about Solr and would allow them to confidently handle 95% of Solr OPs). Is also a good stepping stone to learning Solr
- Adding tests around Solr-API service
- BOR + Business Search have tests that verify the search list ordering, as well as updating, reindexing, updating synonyms, etc.; this allows us to upgrade their Solr versions with confidence

For all Solr projects
- Automate deployment
- Optimize reindexing process and reduce frequency
- Build out solr monitoring + automated scaling and redeployment
- Optimizing GCP infrastructure to reduce costs
- Optimizing Solr queries + managed schemas

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.