PostHog / PostHog/posthog

S3Client missing requestHandler timeout configuration for external S3 storage

Open
#45,972 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Bug Description

Bug Description

When using external S3-compatible storage (like Wasabi) for session recordings, uploads fail with RequestTimeout errors even when SESSION_RECORDING_V2_S3_TIMEOUT_MS is set to a high value (e.g., 600000ms).

How to reproduce

Connect an external S3 to a self-hosted PostHog

Root Cause

The SESSION_RECORDING_V2_S3_TIMEOUT_MS environment variable is correctly read by the config (verified via node -e "require('/code/nodejs/dist/config/config.js').defaultConfig.SESSION_RECORDING_V2_S3_TIMEOUT_MS"), but it's only used for a JavaScript setTimeout in s3-session-batch-writer.js, NOT for the AWS SDK's socket/request timeout.

In /code/nodejs/dist/session-recording/consumer.js (lines 78-89), the S3Client is created without a requestHandler:

const s3Config = {
    region: hub.SESSION_RECORDING_V2_S3_REGION,
    endpoint: hub.SESSION_RECORDING_V2_S3_ENDPOINT,
    forcePathStyle: true,
};
// ...
s3Client = new client_s3_1.S3Client(s3Config);

The AWS SDK v3 has its own default timeouts which are too short for external S3 services with higher latency.

Error Logs

"error":{"name":"RequestTimeout","$fault":"client","$metadata":{"httpStatusCode":400,"attempts":3,"totalRetryDelay":28},"message":"Your socket connection to the server was not read from or written to within the timeout period"}
"totalBytes":5595258,"msg":"session_batch_recorder_flush_error"
Note: totalRetryDelay:28 (28ms) indicates the SDK's internal timeout is very short.

Expected Behavior

SESSION_RECORDING_V2_S3_TIMEOUT_MS should configure the AWS SDK's requestTimeout and connectionTimeout.

Suggested Fix

const { NodeHttpHandler } = require("@smithy/node-http-handler");

const s3Config = {
    region: hub.SESSION_RECORDING_V2_S3_REGION,
    endpoint: hub.SESSION_RECORDING_V2_S3_ENDPOINT,
    forcePathStyle: true,
    requestHandler: new NodeHttpHandler({
        requestTimeout: hub.SESSION_RECORDING_V2_S3_TIMEOUT_MS,
        connectionTimeout: 30000,
    }),
};

Environment

PostHog version: latest (self-hosted)
S3-compatible storage: Wasabi (eu-west-2)
Session recording data size: ~5.5MB batches

Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables or https://eu.posthog.com/settings/project-details#variables]
- [x] PostHog Hobby self-hosted with `docker compose`, version/commit: `posthog/posthog:latest` (image sha256:36eedca28008, built 2026-01-23)
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nodejs/dist/session-recording/consumer.js, where the S3Client is created, and compare it with the timeout usage in s3-session-batch-writer.js and the value exposed by config/config.js. Confirm that SESSION_RECORDING_V2_S3_TIMEOUT_MS reaches the AWS SDK request handler and that external S3 uploads no longer fail with RequestTimeout errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript, node.js
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.