open-telemetry / open-telemetry/opentelemetry-lambda

Node layer never flushes LoggerProvider on invocation completion — logs silently dropped without manual forceFlush()

Open
#2,537 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
432
Forks
246
Avg merge
3d 10h
Merged PRs (30d)
47

Description

Describe the bug
When using the Node layer with AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler and OTLP logs export, logs emitted via the Logs API are unreliably delivered to the backend. They are only reliably delivered if the application explicitly calls forceFlush() on the logger provider before the handler returns. wrapper.ts constructs the LoggerProvider with a BatchLogRecordProcessor but no code path calls forceFlush()/shutdown() on it after the wrapped handler resolves, nor on a Lambda Extensions API SHUTDOWN event. AWS may freeze the execution environment shortly after the handler returns, before the batch processor's timer fires, dropping any log not yet exported.

Steps to reproduce
Deploy a Lambda with the layer below and AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler, OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=<your OTLP endpoint>, using this handler:

import { logs, SeverityNumber } from '@opentelemetry/api-logs'
import { randomUUID } from 'node:crypto'

export async function handler() {
    const experimentId = randomUUID()

    logs.getLogger('otel-logs').emit({
        body: 'Lambda test log',
        severityNumber: SeverityNumber.INFO,
        severityText: 'INFO',
        attributes: {
            experiment_id: experimentId,
        },
    })

    // works only with this line uncommented
    // await (logs.getLoggerProvider() as any).forceFlush?.()

    return {
        experimentId,
    }
}

Invoke repeatedly with the forceFlush() line commented out: log records are missing in the backend intermittently. Uncommenting the forceFlush() line (or alternatively sleeping ~5s before returning) makes delivery reliable.

What did you expect to see?
Logs emitted during a Lambda invocation are flushed/delivered automatically by the layer before the execution environment is frozen, the same "zero code change" guarantee the layer provides for traces.

What did you see instead?
Log records queued in the (default) BatchLogRecordProcessor are lost intermittently, with no application-level indication of the drop.

What version of collector/language SDK version did you use?
Version: layer opentelemetry-nodejs-0_23_0 (arn:aws:lambda:eu-central-1:184161586896:layer:opentelemetry-nodejs-0_23_0:1)

What language layer did you use?
Config: Node.js

Additional context
Relevant code: wrapper.ts constructs the LoggerProvider with a BatchLogRecordProcessor (no schedule-delay override), but no flush/shutdown call exists in the wrapper for the logger (or tracer/meter) provider.

Suggested fix: flush all registered providers (logger, tracer, meter) automatically once the wrapped handler settles, before returning control to the Lambda runtime, and/or register a Lambda Extensions API SHUTDOWN handler to flush on environment termination so in-flight logs aren't lost if the process is killed abnormally.

Workaround: manually call await (logs.getLoggerProvider() as any).forceFlush?.() at the end of every handler.

Possibly related (metrics, not logs, but similar auto-flush gap): #1949

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read nodejs/packages/layer/src/wrapper.ts, starting at LoggerProvider construction and the wrapped-handler completion path; also inspect whether a Lambda Extensions API SHUTDOWN path exists. Reproduce with the provided handler and OTLP endpoint, then verify logs emitted without manual forceFlush() are delivered before invocation completion or shutdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, node.js, typescript
Domain
cloud, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.