elastic / elastic/integrations
[Google Threat Intelligence]: Add data stream for the file feed
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
### Integration Name
Google Threat Intelligence [packages/ti_google_threat_intelligence]
### Dataset Name
New: `ti_google_threat_intelligence.file_feed`
### User Goal
**Problem:** The GTI package covers the GTI threat lists and IOC stream, but has no coverage of the feeds product. The file feed is a different thing from the threat lists: rather than a curated set of indicators, it is a continuous stream carrying a full file report for every file analysed by VirusTotal. Customers licensed for feeds have no supported way to get that data into Elastic today - they have to build a custom pipeline and lose the package's mappings, dashboards and threat enrichment.
**Enhancement:** Add a data stream for the file feed, collectable both via the API and via `filestream` for air-gapped deployments (mirroring the threat list work in #21076).
**A note on naming, because it is confusing:** VirusTotal and Google Threat Intelligence are the same platform. GTI is the commercial brand the product is sold under since Google brought Mandiant and VirusTotal together, and it is the GTI licence tier that carries the feeds entitlement — but the API surface and the documentation are still VirusTotal. So this is both "the VirusTotal file feed" and "the GTI file feed" depending on who is talking. In this package we should keep using GTI for anything user-facing (data stream titles, descriptions, docs) to stay consistent with the package name and with what the customer has bought, and refer to VirusTotal only when pointing at the API or its documentation.
**How the feed works** (https://docs.virustotal.com/reference/file-feed):
- `GET /feeds/files/{time}` where `time` is `YYYYMMDDhhmm` — one batch per minute.
- Each batch is a bzip2-compressed UTF-8 text file, one JSON object per line. Each object is a file object as returned by `/files/{id}`, plus `download_url` and `submitter` context attributes.
- The most recent available batch lags real time by 60 minutes. Batches are retained for 7 days.
- An occasional 404 for a single minute is normal and means no batch was generated — it must be tolerated rather than treated as failure. Consecutive 404s across several batches is a genuine error condition.
- There is also an hourly variant, `GET /feeds/files/hourly/{YYYYMMDDhh}`, returning a `.tar.bz2` of the 60 minutely batches, with a 2 hour lag. Worth considering as the default collection mode — far fewer requests, and the existing threat list data streams already have an `availability_delay` pattern that maps onto this cleanly.
This is not a new auth surface. The feed sits on the same host and uses the same credentials as the existing data streams — the package `url` var already defaults to `https://www.virustotal.com` and the existing threat list streams call `{url}/api/v3/threat_lists/...` with `access_token`. The feed is `{url}/api/v3/feeds/files/{time}` with the same token, so the new data stream can reuse the existing package-level `url` and `access_token` vars as-is.
Feed access requires a specific licence tier, so the data stream should be disabled by default and say so in its description, as the Enterprise-gated threat list data streams do.
### Existing Features
Every data stream in the package is API-only (`input: cel`) against the threat list and IOC stream endpoints. Nothing collects from `/feeds/*`, in any input type.
**Scoping needed before implementation**
- **Volume and filtering.** This is the global stream — every file analysed by VirusTotal, not just the tenant's own submissions — and each record is a full file report including all engine verdicts, not a thin indicator row. Ingesting it wholesale is a substantial and largely low-signal write load. We need a volume estimate from the GTI team and a decision on whether to filter at collection time (minimum detection count, file type, tags) rather than index everything.
- **Mapping.** The threat lists populate `threat.indicator.*` on the basis that everything in them is known-bad. The file feed is mostly benign or unknown, so reusing that structure unfiltered would put a lot of clean files into threat indices and feed them into IOC enrichment. Needs a decision: separate mapping, or `threat.indicator.*` with a filtered subset.
Contributor guide
Assessment
This issue has not been assessed yet.