PostHog / PostHog/posthog

Feature request: aws_ses source, sign POST requests and add the POST-only tables

Open
#86,804 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement feature/pipeline-sources team/data-warehouse
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Feature request

Sign POST requests in the aws_ses client, then add the seven SESv2 read operations that only POST reaches.

Current state: the client is GET-only by construction. send_request builds AWSRequest(method="GET", url=url) and signs it with SigV4Auth, and TRANSPORT_RETRY sets allowed_methods=frozenset(["GET"]). Query parameters go into the URL, and the code encodes them to match the SigV4 canonical query string byte for byte.

SESv2 puts 7 of its 44 read operations on POST, because each one takes a request body. So the client reaches none of them today. This is the reason the source shipped without them, not a scoping preference.

Why now: a customer runs the source against a live AWS account and asks for the widest table coverage we can offer. A separate issue covers the GET-reachable tables.

Every shape below comes from botocore's bundled service model, sesv2/2019-09-27.

Part 1: sign a request body

SigV4Auth.add_auth already signs a body when the AWSRequest carries one. It computes the payload hash and sets x-amz-content-sha256. So the work is to let the client build and send a POST.

  1. Give send_request an optional JSON body, and a method argument that defaults to GET.
  2. Build the AWSRequest with that method and the serialized body. Serialize once, then sign and send the identical bytes. A re-serialization between signing and sending breaks the signature, exactly as a query-string mismatch would.
  3. Set Content-Type: application/json.
  4. Add POST to TRANSPORT_RETRY.allowed_methods. These operations are reads, so a retry is safe despite the verb.
  5. Move NextToken and PageSize into the body for these endpoints. _walk_pages puts them in the query string today, so it needs a per-endpoint switch between query parameters and body fields.
  6. Check the data-imports semgrep transport rules still pass. They constrain how a source builds its session.

Part 2: the tables

Table Path result_key Required input Notes
reputation_entities /v2/email/reputation/entities ReputationEntities none Per-identity and per-configuration-set reputation status. The highest value table in this group for a transactional sender.
import_jobs /v2/email/import-jobs/list ImportJobs none Contact and suppression list import history.
export_jobs /v2/email/list-export-jobs ExportJobs none Metrics and message export history.
tenants /v2/email/tenants/list Tenants none Multi-tenancy. Returns an empty list on an account that uses no tenants.
contacts /v2/email/contact-lists/{ContactListName}/contacts/list Contacts ContactListName Two-level. Fans out from ListContactLists, then paginates inside each list.
tenant_resources /v2/email/tenants/resources/list TenantResources TenantName Two-level. Fans out from ListTenants.
recommendations /v2/email/vdm/recommendations Recommendations none Needs Virtual Deliverability Manager. Report a clear reason when it is off.

contacts and tenant_resources each need pagination inside a fan-out. The current _fanout_page_rows issues exactly one detail request per listed item, so it cannot walk a second NextToken. Extend it, or give these two endpoints their own walk.

ListResourceTenants is the inverse of ListTenantResources. It requires a ResourceArn, and no SESv2 list operation returns the ARNs to feed it. tenant_resources already covers the tenant-to-resource mapping, so leave ListResourceTenants out.

Sync behavior

None of these endpoints accepts a server-side time filter, so every table stays full refresh. Keep sort_mode="desc".

contacts is the one table that can grow large, because it holds one row per subscriber. Confirm the resume path works there. A saved page token inside a fan-out must not restart the outer walk.

Definition of done

  1. The client signs and sends a POST with a request body, and a unit test asserts the signature over that body.
  2. The seven tables appear in the schema picker and sync rows against a live AWS account.
  3. canonical_descriptions.py and ENDPOINT_DESCRIPTIONS carry a description for every new table and column.
  4. probe_endpoint_permissions reports a clear reason for each new table when the IAM policy denies it, and when VDM is off.
  5. SOURCES.md and the posthog.com source doc report the new tables.

Implementation guide: .agents/skills/implementing-warehouse-sources/SKILL.md.

Related: #86803 covers the GET-reachable tables.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .agents/skills/implementing-warehouse-sources/SKILL.md, then inspect send_request, TRANSPORT_RETRY, _walk_pages, and _fanout_page_rows. Add focused coverage for body signing and nested pagination before checking the seven tables, descriptions, permission reporting, source docs, and resume behavior against the stated definition of done.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
api, backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
34/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.