google / google/secops-wrapper

`ingest_log` silently drops events for AIDE (and likely other file-integrity log types)

Open
#213 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
92
Forks
48
PR merge metrics
No merged PRs in 30d

Description

# `ingest_log` silently drops events for AIDE (and likely other file-integrity log types) — `sourceFilename` field missing from `LogsInlineSource` payload

## Summary

`ChronicleClient.ingest_log(...)` constructs a `LogsInlineSource` payload for `POST .../logTypes/{log_type}/logs:import` without the `sourceFilename` field. For at least the `AIDE` log type, Chronicle's ingestion pipeline silently drops submissions that omit it — the REST call returns HTTP 200 with an empty `{}` body (the standard queued-acknowledgement response shape), but the event never reaches the parser and never surfaces in `search_udm`. No error is ever surfaced to the caller.

The REST reference at [`projects.locations.instances.logTypes.logs/import#logsinlinesource`](https://cloud.google.com/chronicle/docs/reference/rest/v1alpha/projects.locations.instances.logTypes.logs/import#logsinlinesource) shows `sourceFilename` as a field on `LogsInlineSource` (sibling of `logs[]` and `forwarder`). The SDK does not expose it — `source_filename` / `sourceFilename` appears zero times in `src/secops/chronicle/log_ingest.py`, and `ChronicleClient.ingest_log`'s signature has no corresponding parameter.

## Environment

- `secops` 0.42.0 (also verified against 0.43.0 — no change to `log_ingest.py` between versions)
- Tenant: Chronicle instance in region `europe`
- Python 3.12
- Authenticated as a service account with `roles/chronicle.admin`

## Reproduction

```python
from secops import SecOpsClient

client = SecOpsClient().chronicle(
customer_id="...", project_id="...", region="europe",
)

# AIDE sample — valid syslog RFC5424 line, well-formed, UTF-8 clean.
# Parses to exactly 1 UDM event via run_parser against the committed parser.
log = '<133>1 2026-04-19T06:04:02.070299+00:00 host aide 1234 - - file=/etc/shadow;Mtime_new=...'

resp = client.ingest_log(
log_type="AIDE",
log_message=log,
labels={"source": "smoketest", "smoketesteventid": "abcd1234"},
)
# resp == {} — reported success
# Subsequent search_udm for smoketesteventid="abcd1234" → zero hits, indefinitely.
```

The submitted event never surfaces. No exception raised, no indication anything went wrong. Reproduced across > 5 probes over several hours, including with uniquely-generated synthetic content (rules out deduplication).

Production forwarder-pushed AIDE events continue to ingest and parse normally (> 10,000/day on the same tenant), so it is not a parser or tenant-level issue — it's specific to the `logs:import` inline path.

## Root cause

The body emitted by `ingest_log` (per `src/secops/chronicle/log_ingest.py` lines 878–886 in the current main) looks like:

```python
payload = {
"inline_source": {
"logs": [
{
"data": "",
"log_entry_time": "...",
"collection_time": "...",
"labels": {"source": {"value": "smoketest"}, ...}
}
],
"forwarder": "projects/.../forwarders/",
}
}
```

There's no `sourceFilename` set at the `inline_source` level. Chronicle accepts the call (HTTP 200) but doesn't process the event.

## Fix (what we had to do)

We bypassed `ingest_log` and built the request directly via `chronicle_request`, setting `sourceFilename` at the correct level — sibling of `logs[]` and `forwarder`:

```python
from secops.chronicle.utils.request_utils import chronicle_request
import base64

payload = {
"inline_source": {
"logs": [
{
"data": base64.b64encode(log.encode("utf-8")).decode("ascii"),
"log_entry_time": ts, "collection_time": ts,
"labels": {"source": {"value": "smoketest"}, ...},
}
],
"forwarder": "projects/.../forwarders/",
"sourceFilename": "smoketest/AIDE/abcd1234.log",
}
}
chronicle_request(client, "POST", "logTypes/AIDE/logs:import", json=payload)
```

With this change the AIDE event lands immediately and appears in `search_udm` within the normal sub-minute window.

Two important placement notes from trial-and-error:

- `sourceFilename` belongs at `inline_source` level (sibling of `logs` + `forwarder`), per the REST reference.
- Placing it inside an individual log entry (`inline_source.logs[0].sourceFilename`) returns `HTTP 400: Unknown name "sourceFilename" at 'inline_source.logs[0]': Cannot find field.`

We separately verified that adding `sourceFilename` to a log type that was previously ingesting fine (`GOANYWHERE_MFT` in our case) does not break anything — it's safe to set unconditionally. (extensively tested /s - Matt)

## Suggested fix for the SDK
(AI generated, apply salt liberally - Matt)

Add a `source_filename: str | None = None` parameter to `ChronicleClient.ingest_log()` and plumb it through to the payload:

```python
def ingest_log(
self, log_type, log_message, ..., source_filename: str | None = None,
):
...
inline_source = {
"logs": logs,
"forwarder": forwarder_resource,
}
if source_filename is not None:
inline_source["sourceFilename"] = source_filename
payload = {"inline_source": inline_source}
```

Optionally, defaulting it to a generated value (e.g. `sdk-.log`) rather than omitting it would make the SDK robust against the undocumented server-side requirement — AIDE and possibly other file-integrity / file-aware log types appear to require it, and users hitting the silent-drop behaviour have no feedback mechanism pointing at the field.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.