Kuadrant / Kuadrant/dns-operator

Large memory usage on Primary clusters

Open
#707 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
12
Forks
23
Avg merge
1d 4h
Merged PRs (30d)
14

Description

**Describe the bug**
With the multicluster-runtime, the events for all the dnsrecords are watch by a primary cluster in a multi cluster set up.
These are cached, causing the memory usage to increase.
Records from the secondary that are not delegated will also be cached.
This cause a high usage requirement on the primaries.

**To Reproduce**
Steps to reproduce the behavior:
1. Deploy a multi cluster set up. `make multicuster-local-setup`
2. Set up coredns secret as the default secret on the primary cluster.
3. Add dnsrecords to the secondary with `delegation=true`.
4. Monitor the memory usage of both primary, and secondary controllers.
5. Expect to see the primary use about double the memory.

**Expected behavior**
Reduce the memory over head on the primary.

**Scenarios**
Some scenarios that should to be investigated

- Multi secondary clusters with a spread of dnsrecords, compared to one secondary with the same number of dnsrecords.
- Multi primary clusters with dnsrcords on the primaries
- What affect does adding secondaries have to the primary if there is no dnsrecords. Is there an affect on the base level resources required per connection.
- What affect does a mixture of delegated, and non delegated have.
- Can a running primary with a high memory usage be restarted. What would the memory usage do during upgrade.
- With a large number of records on secondaries. How long does a reconcile take compared to non delegated records.

**Expected Outcomes**
- Issue to document the findings
- Possible follow up issue with improvement.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the multi-cluster setup with `make multicuster-local-setup`, then monitor primary and secondary controller memory while varying delegated records, secondary clusters, and record counts. Document the observed memory and reconcile-time findings, including restart behavior, and use them to identify a possible follow-up improvement issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.