elastic / elastic/integrations
[CDR] Set transform destination indices to multiple primary shards to avoid shard-size warnings at scale
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
## Motivation
Follow-up to #20820 (Wiz), which flagged that CDR transform "latest" destination indices are created with a **single primary shard** and hit Elasticsearch shard-size warnings at scale (one Wiz customer's vulnerability index reached ~76 GB on one shard, well above the ~50 GB per-shard ceiling).
The parent issue's action item — *"audit other 3rd party integration transforms (Qualys, Tenable, AWS Security Hub) for the same single-shard issue"* — named those three as examples, not an exhaustive list. The real scope is **every CDR transform**, i.e. any transform whose `dest.index` starts with `security_solution-`. This issue tracks the audit and remediation for all of them (Wiz excluded — tracked in #20820).
### Audit result
Filter used: transforms with `dest.index` matching `security_solution-*`. **All 13 (across 11 packages) currently produce single-shard destination indices.** Two failure modes:
1. **No `manifest.yml`** next to `transform.yml` → Fleet defaults the destination index to `number_of_shards: 1`.
2. **A `manifest.yml` exists but only sets mappings / `total_fields` limits — it never sets `number_of_shards`** → still defaults to 1 shard. These packages *look* covered but aren't.
| # | Package | Transform | `manifest.yml`? | Sets `number_of_shards`? | Current `dest.index` | `fleet_transform_version` |
|---|---|---|---|---|---|---|
| 1 | `aws` | `latest_cdr_misconfigurations` | ❌ none | ❌ | `security_solution-aws.misconfiguration_latest-v3` | 0.4.0 |
| 2 | `aws` | `latest_cdr_misconfigurations_awsconfig` | ❌ none | ❌ | `security_solution-awsconfig.misconfiguration_latest-v1` | 0.2.0 |
| 3 | `aws` | `latest_cdr_vulnerabilities_awsinspector` | ❌ none | ❌ | `security_solution-awsinspector.vulnerability_latest-v2` | 0.4.0 |
| 4 | `aws_securityhub` | `latest_cdr_vulnerabilities` | ✅ | ❌ (`total_fields` only) | `security_solution-aws_securityhub.vulnerability_latest-v2` | 0.3.0 |
| 5 | `cloud_security_posture` | `misconfiguration` | ✅ | ❌ | `security_solution-cloud_security_posture.misconfiguration_latest-v1` | 0.3.0 |
| 6 | `eset_protect` | `latest_cdr_vuln` | ❌ none | ❌ | `security_solution-eset_protect.vulnerability_latest-v2` | 0.1.0 |
| 7 | `google_scc` | `latest_cdr_misconfigurations` | ❌ none | ❌ | `security_solution-google_scc.misconfiguration_latest-v1` | 0.2.0 |
| 8 | `google_scc` | `latest_cdr_vulnerabilities` | ❌ none | ❌ | `security_solution-google_scc.vulnerability_latest-v1` | 0.2.0 |
| 9 | `m365_defender` | `latest_cdr_vulnerabilities` | ❌ none | ❌ | `security_solution-m365_defender.vulnerability_latest-v3` | 0.4.0 |
| 10 | `microsoft_defender_endpoint` | `latest_cdr_vuln` | ❌ none | ❌ | `security_solution-microsoft_defender_endpoint.vulnerability_latest-v3` | 0.4.0 |
| 11 | `qualys_vmdr` | `latest_cdr_vulnerabilities` | ❌ none | ❌ | `security_solution-qualys_vmdr.vulnerability_latest-v1` | 0.2.0 |
| 12 | `rapid7_insightvm` | `latest_cdr_vulnerabilities` | ❌ none | ❌ | `security_solution-rapid7_insightvm.vulnerability_latest-v3` | 0.2.0 |
| 13 | `tenable_io` | `latest_cdr_vulnerabilities` | ❌ none | ❌ | `security_solution-tenable_io.vulnerability_latest-v2` | 0.2.0 |
**Explicitly excluded (not CDR):** `qualys_gav/latest_asset` (dest `logs-qualys_gav_latest.dest_asset-2`) and `aws_securityhub/latest_findings` (dest `logs-aws_securityhub_latest.dest_finding-1`) are `logs-*` "latest" transforms, not `security_solution-*` CDR indices, so they're out of scope for this issue.
## Approach
Identical to the Wiz fix (#20820). `number_of_shards` is an **immutable, create-time** index setting, so three changes are required per transform for the new shard count to actually take effect on existing installs:
1. **Add / extend `manifest.yml`** with `destination_index_template.settings.index.number_of_shards: N`. For packages that already have a `manifest.yml`, merge `number_of_shards` into the existing `settings.index` block — do **not** drop the existing mappings/`total_fields` config.
2. **Bump the `dest.index` `-vN` suffix** in `transform.yml` (keep the alias + `move_on_creation: true`). Fleet keeps the existing destination index on upgrade and only re-shards a *new* index — a suffix bump forces creation of a fresh index from the updated template, which the transform then repopulates and the alias moves onto.
3. **Bump `_meta.fleet_transform_version`** (minor) so the transform object is reinstalled with the new `dest.index`.
### Shard count
Wiz standardised on `number_of_shards: 3` based on the observed 76 GB single-shard index. We don't have equivalent per-package scale telemetry, so `3` is a reasonable consistent default (25 GB/shard at 76 GB) — clears the warning for realistic sizes without meaningfully over-sharding small tenants. Adjust per package during review if there's data suggesting a given source stays small (`2`) or runs much larger.
## Definition of Done
For each transform below: add/extend `manifest.yml` with `number_of_shards`, bump the `dest.index` suffix, and bump `fleet_transform_version`.
- [ ] **`aws` / `latest_cdr_misconfigurations`** — add `manifest.yml`; `-v3` → `-v4`; version 0.4.0 → 0.5.0
- [ ] **`aws` / `latest_cdr_misconfigurations_awsconfig`** — add `manifest.yml`; `-v1` → `-v2`; version 0.2.0 → 0.3.0
- [ ] **`aws` / `latest_cdr_vulnerabilities_awsinspector`** — add `manifest.yml`; `-v2` → `-v3`; version 0.4.0 → 0.5.0
- [ ] **`aws_securityhub` / `latest_cdr_vulnerabilities`** — add `number_of_shards` to existing `manifest.yml`; `-v2` → `-v3`; version 0.3.0 → 0.4.0
- [ ] **`cloud_security_posture` / `misconfiguration`** — add `number_of_shards` to existing `manifest.yml`; `-v1` → `-v2`; version 0.3.0 → 0.4.0
- [ ] **`eset_protect` / `latest_cdr_vuln`** — add `manifest.yml`; `-v2` → `-v3`; version 0.1.0 → 0.2.0
- [ ] **`google_scc` / `latest_cdr_misconfigurations`** — add `manifest.yml`; `-v1` → `-v2`; version 0.2.0 → 0.3.0
- [ ] **`google_scc` / `latest_cdr_vulnerabilities`** — add `manifest.yml`; `-v1` → `-v2`; version 0.2.0 → 0.3.0
- [ ] **`m365_defender` / `latest_cdr_vulnerabilities`** — add `manifest.yml`; `-v3` → `-v4`; version 0.4.0 → 0.5.0
- [ ] **`microsoft_defender_endpoint` / `latest_cdr_vuln`** — add `manifest.yml`; `-v3` → `-v4`; version 0.4.0 → 0.5.0
- [ ] **`qualys_vmdr` / `latest_cdr_vulnerabilities`** — add `manifest.yml`; `-v1` → `-v2`; version 0.2.0 → 0.3.0
- [ ] **`rapid7_insightvm` / `latest_cdr_vulnerabilities`** — add `manifest.yml`; `-v3` → `-v4`; version 0.2.0 → 0.3.0
- [ ] **`tenable_io` / `latest_cdr_vulnerabilities`** — add `manifest.yml`; `-v2` → `-v3`; version 0.2.0 → 0.3.0
Per package:
- [ ] Bump the package version in `manifest.yml` and add an `enhancement` `changelog.yml` entry describing the shard-count change and destination-index re-creation.
- [ ] `elastic-package check` / `elastic-package lint` pass.
- [ ] New destination indices are created with the configured primary shards (verify via `GET /_settings`), with aliases pointing at the new indices.
Example `settings` block to add/merge:
```yaml
destination_index_template:
settings:
index:
number_of_shards: 3
```
## Out of scope
- Shard counts on ingest data stream indices (`logs-*`).
- Non-CDR "latest" transforms (`logs-*` dest indices, e.g. `qualys_gav/latest_asset`, `aws_securityhub/latest_findings`).
- The Wiz package (tracked in #20820).
Contributor guide
Assessment
This issue has not been assessed yet.