Strange behaviour after fetching 1MB file from S3 in Lambda function (Node.js)

Open
#485 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws, docker, javascript, node.js
Domain
cloud, devops, infrastructure

Research direction

Start with docker-compose.yml and the LocalStack 0.9.0 configuration, especially the Docker-based Lambda executor, mounted data directory, and Docker socket. Reproduce the repeated S3 and Lambda workflow on macOS, inspect the timeout behavior, and consider the issue done when the bucket can be created after repeated cleanup cycles without restarting Docker.

Written by the indexing model from the issue text.

Description

question

Steps:

  1. Create a bucket
  2. Deploy Lambda function (~20MB) with ffmpeg
  3. Call ffprobe using S3 signed URL:
const input = s3.client.getSignedUrl('getObject', {
    Bucket: 'my-bucket',
    Key: 'video_1mb.mp4',
    Expires: 1000
})

const metadata = await (new Promise((resolve, reject) => {

    ffmpeg.ffprobe(input, async (err, data) => {

      if (err) {

        return reject(err)
      }

      resolve(data)
    })
}))
  1. Stop & Destroy all containers and images
  2. Remove Localstack directory ./localstack
    // Repeat 1 - 5 (~5 times)
  3. Start localstack
  4. Try to create the S3 bucket
  5. Got a timeout error:

make_bucket failed: s3://my-bucket Read timeout on endpoint URL: "http://localhost:4572/my-bucket"

Workaround: Restart Docker daemon

docker-compose.yml

version: '3.7'

services:
  localstack:
    image: localstack/localstack:0.9.0
    container_name: localstack
    ports:
      - '4567-4593:4567-4593'
      - '8080:8080'
    environment:
      - SERVICES=iam,lambda,s3,dynamodb,apigateway
      - DEBUG=yes
      - DATA_DIR=/tmp/localstack/data
      - PORT_WEB_UI=8080
      - LAMBDA_EXECUTOR=docker-reuse
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - './localstack:/tmp/localstack'
      - '/var/run/docker.sock:/var/run/docker.sock'

Host platform: MacOS
Localstack image version: 0.9.0
Docker version: 18.09.2, build 6247962
docker-compose version: 1.23.2, build 1110ad01

Dominant language
JavaScript
Stars
1.4k
Forks
185
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from motdotla/node-lambda

All issues in motdotla/node-lambda

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.