elastic / elastic/connectors

[Connectors Python] Trim outlook messages to body and minimal headers before indexing

Open
#4,033 0 comments 0 reactions 1 assignee Claimed by @Jan-Kazlouski-elastic View on GitHub
effort:medium enhancement outlook priority:medium team:extract-and-transform
Dominant language
Python
Stars
133
Forks
205
Avg merge
16h 3m
Merged PRs (30d)
102

Description

### Problem Description

The Gmail connector used to ship the entire raw RFC 822 message in `_attachment`, which led to large payloads and noisy indexed text in `attachment.content` when content extraction ran (see [#1369](https://github.com/elastic/connectors/issues/1369)). That was addressed for Gmail in [#4031](https://github.com/elastic/connectors/pull/4031).

The Outlook connector should offer the same **body-first, header-light** indexing behavior so mail is usable for search and ML pipelines without dumping routing headers, duplicate MIME parts, or other non-body noise.

Today, Outlook mail documents expose body text via `message` (`html_to_text` on the EWS body) and do not follow the Gmail trimming model. Users who rely on ingest pipelines (attachment processor / text extraction) still need Outlook to behave consistently with Gmail after [#4031](https://github.com/elastic/connectors/pull/4031).

### Proposed Solution

Port the **semantics** of [#4031](https://github.com/elastic/connectors/pull/4031) to the Outlook connector (implementation may differ because Outlook uses Exchange Web Services rather than Gmail’s raw MIME API):

1. **Default (trimmed) behavior** — Index mail content suitable for extraction: prefer plain text over HTML where both exist; drop binary/inline parts from what is sent for extraction; keep only a small, fixed set of headers aligned with Gmail:
`Subject`, `From`, `Reply-To`, `To`, `Cc`, `Bcc`, `Date`, `Message-ID`.
2. **Configuration toggle** — Add `include_full_raw_message` (or Outlook-equivalent label/tooltip), default `false`, matching Gmail’s “Index full raw email (including headers)” escape hatch for edge cases.
3. **Safety** — On trim/build failure, log a warning and fall back to current behavior; no message should be dropped solely because parsing failed.
4. **Reuse where possible** — Share trimming logic with Gmail (e.g. common helper module) if the rebuilt artifact is still a minimal `.eml`; otherwise document why Outlook uses an equivalent representation.
5. **Tests** — Unit tests comparable to `tests/sources/test_gmail.py` (multipart alternative, HTML-only, attachments/inline images, encoded subjects, empty body, toggle on, fallback).
6. **Docs / Kibana** — Update connector configuration reference and Kibana native connector fields if a new toggle is added.

### Alternatives

- **Status quo** — Keep only `html_to_text(mail.body)` on the `message` field. Rejected for parity with Gmail and for pipeline users who expect the same cleaned extraction story as [#4031](https://github.com/elastic/connectors/pull/4031).
- **Pipeline-only fix** — Rely on Elasticsearch attachment processor / Tika without connector-side trimming. Rejected in [#1369](https://github.com/elastic/connectors/issues/1369) discussion; connector-side trimming is the agreed approach for Gmail.

### Additional Context

- Parent: [#1369](https://github.com/elastic/connectors/issues/1369) (Gmail email cleaning)
- Reference implementation: [#4031](https://github.com/elastic/connectors/pull/4031) (`_extract_body_eml`, `include_full_raw_message`, `_message_doc` fallback)
- **Behavior change:** After implementation, upgraded Outlook indices may see different mail body/header content unless `include_full_raw_message` is enabled (mirror Gmail release note from #4031).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.