bcgov / bcgov/entity

Names Solr Service – Post‑Migration Maintenance Plan

Open
#32,162 0 comments 0 reactions 0 assignees View on GitHub
SRE techdebt
Dominant language
JavaScript
Stars
23
Forks
62
Avg merge
24m
Merged PRs (30d)
1

Description

## 1. Overview
This recommended maintenance plan outlines how the Names Solr service should be supported, monitored, and maintained following its recent migration. The goal is to ensure stability, performance, security, and clear operational ownership.

---

## 2. Recommended Ownership Model

### **Primary Owner: SRE Team**
Responsible for:
- Platform-level maintenance
- Infrastructure, scaling, and monitoring
- Security patches and Solr version upgrades
- Backup verification and disaster recovery processes
- Incident response and operational continuity

### **Supporting Owner: Product Team**
Responsible for:
- Schema updates
- Functional search changes or relevance tuning
- Communicating business requirements related to search

---

## 3. Recommended Maintenance Components

### A. Monitoring & Alerting
Implement and maintain:
- Health checks for Solr cores and service availability
- Index size and growth alerts
- Query performance alerts (high latency, timeouts)
- Error-rate monitoring (5xx, failed updates, schema issues)
- Availability-based alerts tied to SLO standards

Tools:
- Google Cloud Monitoring
- Prometheus/Grafana (if applicable)
- Log-based alerting for Solr warnings/errors

Cadence:
- Weekly dashboard review
- Quarterly alert tuning

---

### B. Backups & Recovery
- Perform regular snapshot backups of Solr cores
- Store backups in resilient cloud storage
- Test restore procedures quarterly to ensure data integrity
- Document recovery runbook for on-call engineers

---

### C. Version Upgrades & Security Patching
- Bi-annual Solr version review cadence
- Apply patches or minor upgrades as they are released
- Evaluate major upgrades once per year
- Monitor:
- Published CVEs
- JVM vulnerabilities
- Container base-image patches

---

### D. Configuration Management
- Maintain Solr configuration in Git
- Enforce environment configuration consistency
- Use automated deployment pipelines for changes

---

### E. Index & Query Optimization
Collaboration between Product + SRE teams:
- Quarterly index performance review
- Optimize segments as needed
- Validate schema changes before deployment

---

### F. Documentation
Maintain and update:
- Architecture diagrams
- Operational runbooks
- Deployment/rollback processes
- Backup & recovery steps
- Ownership and contact lists

Review documentation twice yearly.

---

## 4. Recommended Priority & Timeline

| Phase | Timeline | Actions |
|-------|----------|---------|
| **Phase 1 – Immediate (0–2 weeks)** | High | Implement monitoring, confirm backups, establish ownership, create runbooks |

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.