Winston Instrumentation Race Condition in AWS Lambda Layer

Winston Instrumentation Race Condition in AWS Lambda Layer

Describe the bug

Winston instrumentation fails silently when enabled via OTEL_NODE_ENABLED_INSTRUMENTATIONS in AWS Lambda. Logs are not captured or exported to OTLP endpoints. The root cause is a race condition: Winston instrumentation registers synchronously during init() before LoggerProvider is created asynchronously in wrap(), causing logs.getGlobalLoggerProvider() to return undefined.

Steps to reproduce

1. Deploy AWS Lambda function with Node.js 20.x runtime

2. Attach AWS Lambda Layer: arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-arm64-ver-1-30-2:1

3. Set environment variables:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
OTEL_NODE_ENABLED_INSTRUMENTATIONS=aws-sdk,http,winston
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net
OTEL_EXPORTER_OTLP_HEADERS=api-key=<your-license-key>
OTEL_TRACES_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp

4. Create a Winston logger in your function:

const logger = require('winston').createLogger({
  level: 'info',
  transports: [new winston.transports.Console()]
});

5. Invoke the Lambda function with log statements

6. Check your OTLP backend (New Relic, DataDog, etc.)

What did you expect to see?

Winston logs should be automatically captured by the OpenTelemetry instrumentation and exported to the configured OTLP endpoint with proper trace context propagation.

What did you see instead?

Logs never reach the OTLP backend
No error messages or warnings in Lambda logs

Root Cause

Timing issue in nodejs/packages/layer/src/wrapper.ts:

┌─────────────────────────────────────────────────────┐
│ init() - SYNCHRONOUS                                │
├─────────────────────────────────────────────────────┤
│ createInstrumentations()                            │
│   → includes Winston instrumentation               │
│                                                      │
│ registerInstrumentations({ instrumentations })     │
│   → Winston calls logs.getGlobalLoggerProvider()   │
│   → Returns UNDEFINED ❌                            │
│   → Instrumentation becomes no-op                  │
└─────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────┐
│ wrap() - ASYNCHRONOUS (too late)                    │
├─────────────────────────────────────────────────────┤
│ await initializeProvider()                          │
│   → await initializeLoggerProvider()               │
│   → LoggerProvider created and registered ✅       │
│   → But Winston already failed above               │
└─────────────────────────────────────────────────────┘

Workaround

Disable Winston automatic instrumentation and use manual OpenTelemetryTransportV3:

1.Remove winston from OTEL_NODE_ENABLED_INSTRUMENTATIONS:

OTEL_NODE_ENABLED_INSTRUMENTATIONS=aws-sdk,http

2. Install the transport package:

npm install @opentelemetry/winston-transport

3. Use manual transport in your logger:

const winston = require('winston');
const { OpenTelemetryTransportV3 } = require('@opentelemetry/winston-transport');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(
    winston.format.json(),
  ),
  transports: [
    new winston.transports.Console(),
    new OpenTelemetryTransportV3()  // Logs sent to OTLP ✅
  ]
});

This works because the logger is instantiated after wrap() has already created the global LoggerProvider.

What version of collector/language SDK version did you use?

AWS Lambda Layer: aws-otel-nodejs-arm64-ver-1-30-2
OpenTelemetry Core: v1.30.0 (bundled in layer)
OpenTelemetry API: @opentelemetry/api@1.9.0
OpenTelemetry Winston Instrumentation: included in layer
Winston: ^3.13.0

What language layer did you use?

Node.js 20.x (ARM64 architecture)

Additional context

Environment: AWS Lambda Node.js 20.x, ARM64
OTLP Backend: New Relic (tested with https://otlp.nr-data.net)
Issue Location: opentelemetry-lambda/nodejs/packages/layer/src/wrapper.ts (lines ~100-150)
Related Repository: opentelemetry-js-contrib/packages/instrumentation-winston
Impact: Any Lambda function using Winston logger with automatic instrumentation cannot export logs to observability backends
Suggested Fix: Move Winston instrumentation registration to wrap() function after initializeLoggerProvider() completes

Start in nodejs/packages/layer/src/wrapper.ts around lines 100-150 and trace the synchronous init() path, wrap(), and initializeLoggerProvider(). Verify when Winston instrumentation is registered relative to the global LoggerProvider, then confirm the completed behavior with the AWS Lambda and OTLP reproduction steps described in the issue.