OpenFn / OpenFn/lightning

Accept gzip encoded content on webhooks

Open
#3,937 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

Webhooks currently accept JSON data, which will be sent for processing to the workflow.

But the webhooks do NOT accept gzip-encoded data.

Gzipping is a typical way to compress payloads on the wire.

Note that we're NOT talking about attaching a gzip file and extracting it somewhere. We're just talking about how the request body content is encoded.

But it doesn't work today! We get:

Plug.Parsers.ParseError) malformed request, a Jason.DecodeError exception was raised with message "unexpected byte at position 0: 0x1F"
        (plug 1.18.1) lib/plug/parsers/json.ex:95: Plug.Parsers.JSON.decode/3
        (plug 1.18.1) lib/plug/parsers.ex:341: Plug.Parsers.reduce/8

Reproduction Steps

Use CURL to send a JSON object into your favourite webhook:

curl -v -X POST \
  http://localhost:4000/i/ab1801c2-3af0-40cf-8f0c-b743a0afce9b \
  --json @tmp/payload.json \
  --compressed

The --compressed flag will gzip the body

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the webhook request path and the Plug.Parsers.JSON error shown in lib/plug/parsers/json.ex and lib/plug/parsers.ex, then reproduce it with the provided curl command using --compressed. Done means the gzip-encoded JSON webhook request is accepted and processed without the reported parse error; check for a relevant request test in the repository.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
api, backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.